search

org-roam / org-roam

Rudimentary Roam replica with Org-mode
https://www.orgroam.com
GNU General Public License v3.0
5.53k stars 482 forks source link

(Multithreaded) native modules for performance-sensitive components #2383

Open yberreby opened 1 year ago

yberreby commented 1 year ago

org-roam is great, but slows down noticeably when working with many thousands of nodes. This is particularly unfortunate for very-frequently-used operations such as org-roam-node-find. I think it would be worth exploring the option of writing a native extension that internally uses multi-threading, at least for embarrassingly-parallel tasks such as mapping over all tasks.

My own workflow involves writing a very large number of short org-roam nodes, with rich interlinking. I'm at 1700+ nodes after 7-8 months of daily usage. I love org-roam and I'd like to keep using it in this fashion, which is very effective for me, but the gradual slowdown is already felt and is unlikely to spontaneously get better. And yes, I use nativecomp.

In a more general fashion, I contend that performance is a feature, and that having the option to write parallel, native code and leverage external library ecosystems would be a great way to add features to org-roam. One particularly useful example would be semantic search (can be achieved with a small SBERT model such as all-MiniLM-L6-v2, or the more recent bge-small), or efficient graph algorithms for discovering related notes that may not have direct links between them.

Rust has a particularly good concurrency story, so it seems like a great choice for implementing native modules. ubolonton/emacs-module-rs enables writing Emacs native modules in Rust, and seems to have been used successfully by the elisp-tree-sitter project.

Maybe this would best be implemented as a pluggable, opt-in solution in case the use of a language other than Elisp detracts people from contributing; though in this case, there is the risk of feature divergence.

Even if no new features are implemented in the org-roam trunk, having a clear vision of how to cleanly integrate org-roam with native extensions would be valuable.

I am quite comfortable with Rust, though I am not very knowledgeable in Elisp at the time, nor with the internals of org-roam beyond the documentation & cursory looks at the code. Maybe I'm missing a piece of the puzzle regarding interoperability - happy to get some feedback.

Please check the following:

PRESFIL commented 1 year ago

And yes, I use nativecomp.

Do you use Emacs 29's builtin SQL library. I experienced a speedup after upgrading from Emacs 28 to Emacs 29.

yberreby commented 1 year ago

@PRESFIL I tried Emacs 29 recently, but it breaks my Doom setup, and it actually slowed things down for me.

I believe org-roam uses emacsql, did you do anything special to switch to the native client?

PRESFIL commented 1 year ago

did you do anything special to switch to the native client?

As far as I remember no. See help for org-roam-database-connector variable. For me it's value is 'sqlite-builtin.

yberreby commented 1 year ago

Thanks, I'll give it a shot once Doom has complete compatibility with 29.

This is tangential to my main point, though - (non-built-in) native modules & parallelism.