search

EFLS / zetteldeft

A Zettelkasten system! Or rather, some functions on top of the emacs deft package.
https://efls.github.io/zetteldeft
GNU General Public License v3.0
393 stars 42 forks source link

Feature Request: change to Xapian for faster search #10

Open anthrolisp opened 5 years ago

anthrolisp commented 5 years ago

See https://github.com/hasu/notdeft

EFLS commented 5 years ago

Impressive stuff! And since deft can become quite slow with a large number of files, it would be a welcome change.

However, this would (1) be quite a change and (2) make zetteldeft much more difficult for people to adopt (since installing and configuring notdeft doesn't seem that straightforward).

In any case, I'd like to see someone else try to implement the zettelkasten system in this package. It won't be me, or at least not in the near future.

EDIT: After skimming the documentation, I must say this looks like a really powerful approach. Maybe this could fit in some very long term plans...

anthrolisp commented 5 years ago

I'm sure you are already aware of these, but just thought I would hang them here as a matter of interest to other possibilities. Many thanks in advance!

anthrolisp commented 5 years ago

It seem as though you could run notdeft alongside zetteldeft, and use it as a search engine for the zettelkasten without having to modify anything at all.

EFLS commented 5 years ago

Simply running it alongside deft would indeed be no issue, I think. Just like you can search and move through files in any way you like -- after all, zetteldeft is nothing more than plain text files in a single directory. However, the core functions of zetteldeft will still be using deft in the background, so on that front nothing really changes.

anthrolisp commented 5 years ago

FYI, installing notdeft is very straightforward, so long as you're not running Ubuntu!

apraga commented 4 years ago

@EFLS Can you comment on the performance issue for a large number of files ? Can you give an estimate on the slowness and a threshold for the number of files ? I am quite hyped by the tutorial but would like to know how usable it is :)

Edit I've only found this issue on deft's repo talking about it.

EFLS commented 4 years ago

I have only a little less than 500 notes in my Zetteldeft, but haven't noticed any slowness. What do you consider a large number?

There is an important thing to keep in mind here: the issues with regards to slowness are related to (1) Deft starting up or refreshing its database (and caching titles and content), and (2) incremental search.

Generally speaking, (1) only occurs once after starting up Emacs. And (2) can be circumvented by disabling interactive search (as suggested by jrblevins in the link you suggested).

Zetteldeft only relies on programmatic calls on deft-filter, so I don't expect too much issues with large sets of notes.

EFLS commented 4 years ago

I've been testing Deft with a set of 10000 Markdown files containing over 8 million words (available here) and startup time does indeed increase considerably. The non-interactive search, however, remains relatively responsive. So I don't think there is a very big issue at the moment.

apraga commented 4 years ago

Thank you for the quick and detailed answer !

gklimowicz commented 4 years ago

The main Zettelkasten I use has about 4,000 files. It is slow at startup in the standard config. I have found that setting the deft-file-limit, though, works great. It only limits the number of files being displayed at a time, and greatly improves response times. This is what is in my use-package deft :config:

(setq deft-file-limit 200)

I don't have a reason to look at more than a screenful of candidates, the way I use it.

jd-m commented 4 years ago

Hey, I'm no elisp programmer but I've made a set of functions as a zettelkasten on top of notdeft which deals with the 10000 markdown files with no problem. It's kind of like what EFLS did with deft but its way less polished. But it does have some extra window following features for viewing files in search.

It's not very well documented, I'm hoping to improve that soon but in case anyone is interested. https://github.com/jd-m/notzettel.

EFLS commented 4 years ago

Oh wow, that's awesome. I might have to take a look for inspiration for additional Zetteldeft features!

jd-m commented 4 years ago

Please do! It still needs a lot of work but for me it is usable.

TRSx80 commented 4 years ago

https://github.com/jd-m/notzettel

Am I the only one to think perhaps it should have been called notzetteldeft? :smile:

TRSx80 commented 4 years ago

Thanks to all the people who shared their actual (vanilla) Deft performance (and especially, mitigation options), I no longer think this is so much of an issue as I did when I was first researching.

In addition to all the alternatives anthrolisp mentioned in this post (up thread), I had made a fairly in-depth study of related software (including performance, and other issues) that people might also want to take a look at here.

But the TL;DR at the end of it all is that, for now, I decided just start using regular Zetteldeft (on top of regular, vanilla Deft). I am very confident, however, because I learned there are a number of additional tools available to grow into should I ever feel the need. For example, depending on what I found lacking, I would probably progress to incorporating one or more of the following tools, any of which can easily be added alongside / in concert with my current setup:

All of the above mentioned packages have the advantage (unlike notdeft) of being readily available as packages, either on (M)ELPA in the case of Emacs stuff, or distribution package managers in the case of Recoll. No need of compiling or any other dickering around. They also are more generally useful outside of strictly Zetteldeft context.

In fact I am pretty sure I am going to start using all three of them, sooner or later, whether with Zettel/Deft or not!

Cheers! :beers: