search

mazed-dev / truthsayer

Second brain for knowledge workers that retains everything you read online and helps you to back up your communication with your source as easily as you might share an anecdote over a coffee.
MIT License
2 stars 2 forks source link

research how client-side search can be improved #108

Open SergNikitin opened 2 years ago

SergNikitin commented 2 years ago

At the time of this writing our search is implemented as follows (conceptually at least, since encryption isn't enabled yet):

  1. User enters a search query that is compiled into a dumb case insensitive regex
  2. Encrypted data is retrieved from smuggler in batches using TNodeSliceIterator (see #105)
  3. Client application decrypts the data
  4. Regex is applied to contents & search index of each decrypted node

Naive regex-based solutions are incompatible with more advanced search capabilities that we are aiming for (like lemmatization, search by image color etc.)

We need something more powerful and flexible. On server-side there are well-known behemoths like Elasticsearch, but we need something that's pure client-side and is built on top of TNodeSliceIterator, same as before. It would be great if there is something out there that can be plugged into truthsayer as an out of the box solution to kickstart our efforts. If there is nothing out there then we can write our own, but I suspect there should be some basic search building blocks out there as JS/TS libraries.

Part of Thread-knowledge/smuggler#22

akindyakov commented 2 years ago

User enters a search query that is compiled into a dumb case insensitive regex

For this see method makePattern in truthsayer/src/grid/SearchGrid.tsx

akindyakov commented 2 years ago

Renamed it a bit just to clarify a bit - the regexp is fine for now, just not enough. So we are ok if a new search includes regex (or not), the point is to improve it with the data we have for nodes today

akindyakov commented 2 years ago

An "archive" feature is related to this one as a way to surface "dirty" nodes in search results #23