Add full text search

[thombruce]

Not entirely certain how this will work with LokiJS...

Looks as though some docs were in the process of being written here: https://github.com/LokiJS-Forge/LokiDB/blob/docs/docs/full-text-search.md

If the docs there are accurate, search needs to be implemented at the time of collection creation. This may require migration logic, if Toodles is already in use.

---

Gives me pause to wonder again whether LokiJS is the correct solution. Yes, it's a powerful and mature solution, but the lack of recent development sort of puts an end date on its usability. It would be awkward to remain committed to LokiJS if something more appropriate were to come along...

[thombruce]

Removing LokiJS would be a massive refactor, but best done early. The in-memory component could return to Pinia. Persistence could be handled by several libraries - Dexie is one I've used before for IndexedDB.

I guess I'll be looking at JS database solutions again...

Must have a solution for in-browser (e.g. Dexie) and a plan at least for file-system storage and native storage on desktop and mobile.

[thombruce]

PouchDB remains a stand-out option: https://github.com/pouchdb/pouchdb

I do worry about then being committed to a server-solution like CouchDB... but maybe that's a good thing?

[thombruce]

If we're switching persistence solutions. Do also consider what is desirable on a server.

I would usually opt for Postgres... but Mongo and Couch are worthy of consideration.

This decision may impact the choice of client-side solution.

I've discussed elsewhere an intention to target MongoDB, as I feel it may be the most open source developer-friendly. There's no direct equivalent in terms of browser-databases; we'll probably enqueue sync jobs during offline use.

If Mongo is the target, we just need to consider what a solution looks like for...

1. The browser (probably some IndexedDB wrapper like DexieJS)
2. The desktop/Electron (we can theoretically bundle almost any solution we want - something that targets saving to the filesystem)
3. Mobile

Solutions may differ, each being the "local adapter" which would be saved to before a syncing job is enqueued.

Since we only need to be concerned with the browser environment for the time being, is there any reason not to use Dexie?

A popular alternative is LocalForage - https://github.com/localForage/localForage - but this appears to be more of a key/value store, whereas what I'm looking for is something more document-focused.

I think LocalForage would be ideal if we were storing the raw todo.txt-like string (one line in a document), but Dexie probably wins this... and Dexie has pretty good search documentation; starting here: https://dexie.org/docs/Samples

[thombruce]

Dexie's latest update was yesterday, so no worries about it being out-of-date or abandoned.

We'll target a different solution per-environment (something we already were going to do for server vs everything else).

[thombruce]

Still undecided today, but options under consideration are:

• Dexie
• Pouch
• IndexedDB (as-is)

We probably want to use one of the first two, but it's a good idea to look at the API for IDB and what's possible... Maybe we can find a decent full text search implementation that works directly with it.

[thombruce]

See note 6 under how Dexie differs from other wrappers: https://dexie.org/docs/Dexie.js.html

That's a very nifty feature, and one I've been sort of hacking together with map functions...

We probably go Dexie.

[thombruce]

Whatever way we go about implementing full text search, we're likely gonna want a "words" index in the store. Example implementations in Dexie all feature this.

I'd be happy avoiding this in the short term... but then the only other alternative is a relatively complex search algorithm that splits items individually per record anyway, and is a costly operation. Probably better to bite the bullet and create said index early.

We can do this manually, or use something like Lunr or Fuse.js. Lunr hasn't been updated in years but is still recommended by Dexie. No idea what Fuse.js could do for us, but it might be worth a comparison.