improve fulltext search performance

[GoogleCodeExporter]

What steps will reproduce the problem?
1. login to honeycrm, make sure it is populated with some data
2. start the full text search across all modules
3. according to profiling results fulltext search is currently the slowest 
feature in honeycrm since it involves the most db scanning (see attached 
screenshot from profiling session)

What is the expected output? What do you see instead?
we need faster response times to be able to handle at least 10k items per table.

Please use labels and text to provide additional information.

Original issue reported on code.google.com by ingo.jae...@gmail.com on 19 Aug 2010 at 12:41

Attachments:

• fulltext.png

[GoogleCodeExporter]

create a test case for this scenario and optimize it for speed.
we may have to pull full table scans periodically out of the db into memcache 
and feed the full text search from the cached data.

advantages:
speed (?) since full table scans have to be less often (each n minutes or so 
instead of doing full table scans on each full text search request). the 
importance is exaggerated even more since clients to too many requests 
currently.

disadvantages:
data might be out of date
additional complexity. datastore should be fast enough so that this should not 
be necessary.. try client side modifications too to massively reduce server 
load.

Original comment by ingo.jae...@gmail.com on 19 Aug 2010 at 3:14