search

ssbc / ssb-friends

Manages the SSB social graph
MIT License
23 stars 6 forks source link

Support db2 #49

Closed arj03 closed 3 years ago

arj03 commented 3 years ago

This PR adds support for db2 by checking if db2 has been loaded in secret stack and if so uses a level index in the same way it used a flumeview-reduce index before.

In this PR I also took the liberty of doing a bit of cleaning up. I converted the CI to github actions so we have tests running again here. I moved some of the glue code into a separate folder and made it possible disable them. I don't like modules doing this kind of spooky interactions, so I would rather have ssb-friends do 1 thing and that is to give a friends graph and then leave all the glueing of the modules together in another place.

I'll squash the commits for a nicer history, but first I want to put this into ssb-browser and see how it behaves.

arj03 commented 3 years ago

Dictionary compression test seems to be working a bit better. Test from my browser profile seeing roughly 1000 feeds at hops 2.

total data 1148126
getting contacts state: 97.003ms
feeds: 10035
parsing contacts state: 41.602ms
json size of feeds directly 561960
json size of contacts 3978489

The last 2 numbers are only for comparison. What they mean is that the dictionary compression got the size down from roughly 4mb json to 1.1mb. Still level is wierd:

36M /tmp/ssb-browser-sync/db2/indexes/contacts

This is not ready for review yet, just dotting down numbers as I get closer.

arj03 commented 3 years ago

Ho ho ho, turns out level is insanely naive when it comes to batches. Massive speedup incoming :)

arj03 commented 3 years ago

This last commit fixes the batching which in turn have several nice benefits:

arj03 commented 3 years ago

After running this in the browser for a bit of testing I discovered a bug. I think the bug is actually in ssb-db2.

arj03 commented 3 years ago

Found the problem. It's a bit wierd that you can shadow a const by doing a let in a different scope :(

KyleMaas commented 3 years ago

How far is this from being ready to go?

arj03 commented 3 years ago

It needs a review and then I'll probably squash the commits before merging

KyleMaas commented 3 years ago

This seems to work quite well in ssb-browser-demo. I've been clicking through it like a madman trying to get it to fail, and it just seems to keep working. And it's way faster than it was with ssb-social-index and ssb-suggest.

arj03 commented 3 years ago

Thanks for the review