Feature Request: Use fuzzy finding for search

[diego898]

I've long thought search should be at the core of TW, and could be greatly improved. One such way is to use fuzzy finding: http://fusejs.io/

I think this would be a great addition to the TW core, and would supersede many attempts by those in the group to improve search.

[diego898]

Or perhaps a workflow similar to this:

https://keminglabs.com/finda/

[Jermolene]

Hi @diego898 thanks fuse.js is interesting. I've seen a few JS search libraries that would be interesting to explore (http://elasticlunr.com/ being one of them). But I think it belongs as a plugin. Note that the diff-match-patch library recently integrated into TiddlyWiki does contain a low-level fuzzy matching implementation as a by-product of the main diffing implementation that we might be able to re-use: https://neil.fraser.name/software/diff_match_patch/demos/match.html)

[diego898]

Thanks for the response @Jermolene! I respectfully push back a bit at making it a plugin (but of course would happily accept one/make one if I could!). If TW really is a "non-linear notebook for capturing, organising and sharing complex information.", I think searching that complex information should be a "first class citizen".

[Jermolene]

Hi @diego898 no problem; the question of whether to make it part of the core is in any case somewhat hypothetical until there is an implementation.

[thediveo]

Working on a fuse.js-powered plugin, https://github.com/TheDiveO/TwFusejs. At the moment I'm exposing fuse.js through a fuse[...] filter operator. It also accepts fuse search options packaged in data JSON tiddlers and then referenced in the operator suffix, such as fuse:myopts[...].

The tiddler fields currently need to be explicitly listed in the fuse search options; I suspect that automatically searching all fields might be possible either by specifying just the fields attribute, or in case this doesn't work, by supplying our own attribute helper function to fuse.js. Some more trial and error necessary here.

Maybe a dedicated <$fuse> widget might be better in some situations, as it could expose multiple variables per result, which is next to impossible with filters. Feedback welcome, PRs more so.

[diego898]

Hey @TheDiveO, just wanted to let you know about @hoelzro's excellent full-text-search plugin:

https://github.com/hoelzro/tw-full-text-search

based on lunr.js. It currently gives us fuzzy matching and much more!

https://github.com/hoelzro/tw-full-text-search/issues/5

[thediveo]

My Fuse.js-based TwFusejs plugin for TW5 has now reached beta status. It's on purpose a lightweight solution, and no one-size-fits-it-all search plugin. As others have pointed out, there are very promising and powerful search plugins when you can live with managing an search index. This plugin aims at the beginners with smaller wikis who are interested in a maintenance-free fuzzy search entry solution.

• Live demo of my TwFusejs plugin
• Plugin description page
• TwFusejs GitHub repo

[thediveo]

As for deploying lunr.js with an automatically maintained index: what about moving lunr into a webworker, and then feeding it tiddler updates, as well as "all" tiddlers on startup? One thing that looks ugly are the way webworkers, erm, work: they get loaded form an URL. Maybe data: URIs would work(*), so a TW5 plugin would need to spawn a webworker that gets lunr fed, together with a shim receiving tiddlers via queue, and updating the index inside that webworker...

(*) at least this SO question indicates that webworkers can be created using a data: URI: https://stackoverflow.com/a/10372280

Useful webworker data URI test on jsfiddle: http://jsfiddle.net/uqcFM/37/

[hoelzro]

@TheDiveO I actually recently added that to TW-FTS; the only part I don't do is the "all tiddlers on startup" part. I haven't added it yet because of lack of time, but also because it hasn't been an issue for me in practice. If there was enough support for that feature, it wouldn't be too hard to add!

[diego898]

While I'd love to see this in the core, @TheDiveO and @hoelzro's plugin get us most of the way there.