Trilium Search Roadmap

[damons]

First: Excellent application and idea.

What are the plans for search? Is there a roadmap?

[zadam]

For 0.31 I work on real fulltext search (using sqlite's FTS5) which allows bunch of advanced things like "find notes with occurences of 'foo', but not 'bar'".

Also filtering based on attributes is being rewritten which will allow stuff like "get me notes created last month".

First beta should available in few weeks.

[damons]

This is excellent news. Thank you.

Are you interested in peer-to-peer search at all? One of the characteristics of Trilium that I enjoy is the concept of secure notes, and I'm working on a p2p secure search engine for documents between trusted clients. Trilium provides a nice interface to about 90% of what a personal search engine interface would look like.

For example, a type of item that could be added under >>root could be a search term. Maybe I'm interested in "Distributed Latent Dirichlet Allocation" so I'd add a >> root >> "Distributed Latent Direchlet Allocation" node/note which is a constantly monitored search term that participates in a p2p search mechanism that searches my unencrypted notes as well as my peers unencrypted note libraries that they've shared with me by way of a previous connection.

Any thoughts?

[damons]

Another piece that I'm integrating into my search engine is the DAT protocol. See https://github.com/datproject . Adding dat:// notes for example leads to an audit trail of edits and revisions. Very handy when deploying distributed collections of notes to peers.

[wesrer]

@damons Sorry if I am interrupting a conversation here, but how is this different from simply caching a copy of shared documents to the local database and then allowing local search?

[damons]

@wesrer Excellent question. The difference is scale. By transferring the compute responsibility out to my peers for search, I can search then entire space in parallel. The more connections I have to my friends' notes, the faster I can search them. Also, when they discover a new document--even if by passively adding it to their own note databases--because they found some article related to a different subject, but it just happens to also contain information about a search term that I've running in my own Trilium, there will be an async update back to me with a notification of a search hit.

The idea is to distribute search amongst all our devices which are constantly connected. Cell phones and everything. I want to do away with the siren server search model (i.e., google, bing, etc.) so that we have personal control over what information we share, especially when it comes to what information I'm searching for on the Web.

Really good question. :)

[damons]

Also, I do want to pull in as much of their note database as possible so I can search locally. However, once I reach my storage load threshold of say, 32G of documents, I require the ability to relieve the pressure by pruning history and relying on my peers who were the source of the original notes/changes/logs, plus all of the peers-of-peers that also contain copies of note histories.

[damons]

The idea is that we build trusted connections to peers, just like we do with all sorts of other P2P applications, be they teachers, students, people who read and search the Web, and slowly replace centralized search.

We hold supercomputers in the palms of our hands, and it's time to leverage those compute and storage resources.

[zadam]

For Trilium Notes sharing notes between people is a secondary concern and currently out of scope. But even my long term plans in this direction are supposed to cover sharing with only couple people and reasonable data sizes so the idea of this P2P search is not something I considered so far and frankly seems to be an overkill for the use cases I want to support with Trilium Notes.

[damons]

Sure thing. Thanks for letting me know. Will offer up anything interesting that emerges.

Thanks for all of your hard work.

Yours faithfully, Damon