Closed d12frosted closed 2 years ago
update: moved release plan to the first message
I am already using this branch and must say... it feels so snappy! Absolutely love it!
I will merge it today closer to evening.
Wow, these performance differences are so amazing! That must be added to org-roam
, please! (And if I have an extra wish, make it accessible with an API so that Delve
does not have to replicate the queries...)
Do I understand correctly that this view table also contains the links?
That must be added to org-roam, please!
I would love to work on that actually. First I wanted to investigate this idea for some time to see if it works. And then ask @jethrokuan if he is willing to accept something like this. But since this conversation started to happen, then... @jethrokuan, what do you think? :)
Do I understand correctly that this view table also contains the links?
Yes. Basically it started from #106. Plus I wanted to provide similar functionality (e.g. to search for links with N
depth) for https://github.com/d12frosted/vino library. For now you can use vulpea-db-query-by-links-some
and vulpea-db-query-by-links-every
- it allows to search for depth = 1. I want to spend some time to optimize more deep search.
BTW, inclusion to org-roam
would easily solve write performance.
Release plan:
After some time (e.g. maybe one week or more), I release new version of
vulpea
.I am still not happy with performance query functions on a set of 9k+ notes. Partially, the culprit here is that horrific SQL. The more tables I want to multiply, the worse performance becomes.
So idea here is simple. Instead of doing multiplication of matrices during query operation, use a view table where every row is a full
vulpea-note
(in reality multiple notes because of aliases, but that's irrelevant here). This new view table is used in the following functions:vulpea-db-query
vulpea-db-query-by-ids
vulpea-db-get-by-id
This transitively affects all specialized queries.
The way view table is being built is not optimal as it parses the buffer for the second time in addition to
org-roam
routine. Ideally this feature should be proposed toorg-roam
thus improving build time.Benchamarks
Count of notes: 9554.
General benchmark with regular
org-roam
tablestags-some
tags-every
links-some
links-every
General benchmark with view table
tags-some
tags-every=
links-some
links-every
Comparison of
vulpea-db-query
tags-some
tags-every
links-some
links-every
Since
vulpea-db-query
loads everything into memory, no surprise that improvement is pretty much the same across test scenarios. The good part here is that on average new implementation performs 4.595 times faster. And since it stays around 1 second for 9554 notes it makes it quite practical.Comparison of specialized queries
tags-some
tags-every
links-some
links-every
Since specialized query performance depends on result size (e.g. how many notes it needs to load into memory) the improvement is not stable. But even for big slices it outperforms previous implementation. On average, new implementation performs 4.34 times faster. Not really useful metric in this case, but still.
Comparison of db sync
Now the real question is how synchronisation is affected. We have the following test scenarios.
vulpea-test-notes
small
medium
huge
As you can see, it affects dramatically full rebuild, but when it comes to singular updates, the difference is pretty small.
Conclusion
View table improves query performance dramatically. Since most of the time I am using query functions and reading notes, this small difference on syncing small to medium files is not critical for release of view tables. Though full rebuild penalty is quite big and should improved in the future iterations.
P. S. I am leaving the full sync test case disabled because I don't want to add 8.5 minutes to the tests. If needed, just replace xit with it.