search

couchbaselabs / cbforest

C++ wrapper library around ForestDB, for use in Couchbase Lite.
17 stars 12 forks source link

Reimplement FTS using an external library (SQLite? CLucene?) #61

Open snej opened 8 years ago

snej commented 8 years ago

The full-text search implementation in CBForest is pretty basic. It doesn't have any fancy query features like boolean operators or even word-prefix matching, and it's not very fast. The best way to improve it is probably to replace it with an existing already-proven implementation.

The best choice, ironically, is SQLite. Its FTS4 is pretty good, and is what Couchbase Lite already uses with the default SQLite storage engine. It basically comes for free on iOS/Mac/Android.

Another alternative is CLucene, a C++ port of Lucene. It's rather a lot of code though (about 80,000 lines of C++.)

The result would be that the FTS storage for a view wouldn't be stored in ForestDB but in the external database used by the FTS library. The CBForest API would need to make this transparent so clients can use the same API for queries.

snej commented 8 years ago

SQLite's FTS comes with a pretty basic tokenizer that doesn't handle stop-words or stemming. We've been using the fts3-unicodesn library for that, which incorporates Snowball stemmers for a dozen languages, plus a custom list of stop-words.

It would be good to make this pluggable, so we can use OS facilities instead. For example, the Cocoa Foundation framework has classes for doing tokenization and stemming, which I believe support many more languages, including Asian ones which are very difficult to work with.

snej commented 8 years ago

Here are some performance comparisons between SQLite, a couple of flavors of Lucene, and one I hadn't heard of called Tracker. SQLite and CLucene are roughly equivalent, and since SQLite's effective code size is nearly zero, it seems like the best choice.

snej commented 8 years ago

There's also Xapian, which has a nice feature set.

brendand commented 8 years ago

the Cocoa Foundation framework has classes for doing tokenization and stemming, which I believe support many more languages, including Asian ones which are very difficult to work with.

This would be amazing to have as I have a lot of Asian customers.

lucatorella commented 8 years ago

I'm very interested in this feature, since I'm using couchbase with ForestDB and currently word-prefix matching is not supported. Is there any time frame for implementing this feature?

snej commented 8 years ago

No, it's not on our roadmap so far. Of course I'd be happy to advise & assist someone else who wanted to work on it…

brendand commented 8 years ago

@lucatorella Word-prefix matching is the only reason why I haven't switched to ForestDB just yet.