Lite friends

[arj03]

• maybe no index for contacts, just use jitdb

[arj03]

@staltz notes for tomorrow

Data points:

• This is for 300k contact messages
• existing ssb-friends uses flume-reduce which just stores a dict in a file. Meaning perf gets quite bad over time
• jitdb getcontacts for 1 feed: 250ms
• add contacts to base index:
  • getcontacts for 1 feed: 40ms
  • speed / usage for 1 mill msgs: 145mb, 73 seconds index time
  • speed / usage for 1 mill msgs without contacts: 130mb, 70 seconds index time

[arj03]

I consider contacts quite fundamental to SSB which is why I'm thinking of putting it in core. The overhead if quite minimal. I was looking over ssb-friends and if you rip out the db part, it is mostly convenience methods. So could live in its module and just use db2 as the storage.

[arj03]

Also by doing it this way we don't need to keep very much in memory which I think existing friends does. This should be especially good for startup time.

[staltz]

ssb-friends has two layered reduces: one reduce for "is A following B?" over all contact messages that A created, and another reduce that depends on the previous one: "at what hops distance is C from A?"

[staltz]

• [ ] Figure out which APIs are used/needed (e.g. in Manyverse)
• [ ] Consider the API for forked ssb-friends (e.g. no createLayer)
• [ ] Fork ssb-friends to use db2 APIs
• [ ] Make a compat layer forked ssb-friends for things that ssb-replicate and ssb-ebt need
• [ ] Make a JITDB operator for ssb-friends, e.g. hops(1)

[mixmix]

I thought with Dijkstra that it has a graph and the you can ask it about any start point with that I have only seen one call to createLayer? Do you mean there are two reduces inside layered graph?

[arj03]

We probably want to start with dynamic-dijkstra and just remove the layered graph stuff. As always things like this makes me uneasy.

[arj03]

It turns out my test script for jitdb had quite a bit of overhead, so now its between 40ms for small profiles (staltz phone) to 100ms for mix. In comparison base index is 15ms and 58ms. Also my earlier observation about adding select style queries instead of just returning the whole message still holds. It is not faster.

[arj03]

This task turned into a friends PR. Currently the index format is too slow, but I'm quite sure it will work with dictionary compression. Will try that today.

[arj03]

Now that the db2 compatibility is merged into ssb-friends I'll close this