Improve speed of text highlight.

[fortable1999]

Original report by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).

---

I'm using whoosh as an indexer in my app, it scanns over a mercurial repositories. When marking search results using highlight module sometime in can kill my server. For example building an index on few dictionary files and searching one word from this index makes my app freze for 2-4 minutes off 100% cpu usage to highlight the one word from content.

This is how i do it it's not a complicated analyzer or formatter.

#!python
analyzer = RegexTokenizer(expression=r"\w+") | LowercaseFilter()
formatter = HtmlFormatter('span',
    between='\n<span class="break">...</span>\n') 

#how the parts are splitted within the same text part
fragmenter = SimpleFragmenter(200)
#fragmenter = ContextFragmenter(search_items)

for res in results:
    d = {}
    d.update(res)
    hl = highlight(escape(res['content']), search_items,
        analyzer=analyzer,
            fragmenter=fragmenter,
            formatter=formatter,
            top=5)

It would be awesome if it could be improved.

[fortable1999]

Original comment by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).

---

Wow, I'm impressed by Your work. It worked out very good, first test that i run showed chunked out and highlighted content in fraction of seconds. Build index with those changes is only slightly larger.

Works perfect !

Keep on the good work, whoosh is for me the #1 library i discovered.

Regards

[fortable1999]

Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).

---

Well, I had to fix up the implementation of spans, but this should be a start...

This solution involves storing the characters at which each term occurs in each document as part of the index. Then when you go to search for a query, you have to use the low-level Matcher API to get the whoosh.spans.Span objects representing matches in the document. The spans will give you the start and end character of each occurance in the text, so you can pull out a chunk around each occurance and highlight/display that.

#!python

# Instantiate a new field type that's like TEXT but uses the Characters
# format, which stores the start and end character of each term in the
# index
charfield = fields.FieldType(format=formats.Characters(analyzer),
                             scorable=True, stored=True)
# Use the custom field specification in the schema
schema = fields.Schema(title=TEXT(stored=True), content=charfield)

# Index the collection with this schema
# ...

# Now, to search for my_query.
# Use the lower-level Matcher API instead of a Results object
# to get the matching spans.
searcher = my_index.searcher()
matcher = my_query.matcher(searcher)

# Loop over the matching documents
while matcher.is_active():
    # The document number of the current match
    docnum = matcher.id()
    # How to get the stored fields for the document number
    d = searcher.stored_fields(docnum)
    content = d["content"]

    # Loop over all the places in the document that matched
    # the query
    for span in matcher.spans():
        print "  Match from %d to %d" % (span.startchar, span.endchar)
        print "  ", content[max(0, span.startchar-40) : span.endchar+40]

    # Move to the next matching document
    matcher.next()
searcher.close()

Of course this is not an ideal solution. It's low-level, and for a complex query (e.g. OR with lots of words) you'll get spans all over the place. The Spans class has a couple of simple methods to merge spans (Span.to() and Span.merge()), but it would be nice to integrate spans with highlighting so the highlighting system could find clusters of spans and use them to pick the best "chunks" to highlight. But hopefully this is enough to help you.

[fortable1999]

Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).

---

OK, I get it now. This is definitely possible, by storing character positions in the index, but it will increase the size of the index and requires using lower-level APIs. I'll work up an example and attach it.

[fortable1999]

Original comment by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).

---

My use scenario is a full text search in source codes, and I have to use "heavy" tokenizers, since i would like to search on imports "from whoosh.example import something" which are divided by '.' in python. Other programming languages are totally different so finding such light tokenizer would be difficult.

The problem in my case is that when searching for a specific pattern i accidentally hit a large file, the highlight freezes the application.

The perfect solution for me would be if whoosh could chunk the content for interesting pieces only, similar to SimpleFragmenter, and then highlighting would be super fast. I don't know if such solution is even possible. I image that many of codes would have log files people like to search in my app and those can be big.

[fortable1999]

Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).

---

Yes, tokenizing a 3.5 MB file like that, including all the codes, is just going to be slow in Python. Changing the regular expression to only parse the words at the start of each line at least reduces the time to 3s. What is it you want to do? Were you just using this file as a test?

[fortable1999]

Original comment by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).

---

I don't know much about the internals, but I think it's just to much text to parse, maybe if there where info how to efficient chunk the content just for highlighting it would help allot.

[fortable1999]

Original comment by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).

---

Try opening 'cmu_phonetic_dictionary' i run Your code and got 20s on 3Ghz core2

[fortable1999]

Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).

---

On my first try, I can't reproduce your problem. This is what I tried:

#!python

from whoosh import analysis, highlight

analyzer = analysis.RegexTokenizer(expression=r"\w+") | analysis.LowercaseFilter()
formatter = highlight.HtmlFormatter('span', between='\n<span class="break">...</span>\n') 
fragmenter = highlight.SimpleFragmenter(200)
#fragmenter = ContextFragmenter(search_items)

f = open("context.txt", "rb")
text = f.read().decode("utf8")
f.close()

from whoosh.util import now
from cgi import escape

t = now()
text = escape(text)
hl = highlight.highlight(text, ["abandon", "rocket"],
                         analyzer, fragmenter, formatter, top=5)
print repr(hl)
print now() - t

This prints:

u'Unfortunately, Klietz was eventually forced to <span class="match term0">abandon</span> his work.  The\ncompany that originally owned the rights to Screenplay, Gambit, was\nsubsumed into a larger company, Interplay.  Interplay later filed for'
0.0992741478549

This isn't especially fast, but it's definitely not a freeze. Is there something else I should try?

[fortable1999]

Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).

---

Thanks for your quick reply, I'll check this out.

[fortable1999]

Original comment by Marcin Kuzminski (Bitbucket: marcinkuzminski, GitHub: marcinkuzminski).

---

Here's the files that's causing this. Try to search for example for "abandon rocket" it takes approx 20-30s to highlight those

[fortable1999]

Original comment by Matt Chaput (Bitbucket: mchaput, GitHub: mchaput).

---

That's really strange. Is it possible to attach an example of the text you're highlighting (i.e. an example of res['content'])?