Support group partial replication

[staltz]

• [x] Install ssb-tribes2 in this module
• [x] Read ssb-tribes2 for all the groups you're part of
• [x] And list all group members
• [x] Match each group member against the templates
• [x] Introduce $groupSecret as a magic token when writing the templates
• [x] Implement group key in the templates
• [x] RequestManager refactor to not track hops but instead use a hops cache
• [x] group replication tests pass
• [x] RequestManager should unrequest shards only if all of its children are unrequested
• [x] RequestManager to handle case where main feed is blocked but is a group member (this should be tested!)
• [x] Check code coverage for: RequestManager should unrequest shards only if all of its children are unrequested
• [x] RequestManager seems to be in an infinite (async) loop, see DEBUG=ssb:replication-scheduler and _block()
• [x] Make sure that old previously-encrypted add-member messages don't get missed (some kind of reindexencrypted)

Bonus:

• [ ] MetafeedFinder should minimize how many RPCs are sent out: queue fetch() until _loadAllFromLog is done
• [ ] MetafeedFinder does an addOOO, this could potentially be dangerous, so put a debug log for it and track this in our experiments

[arj03]

So if you specify group you are replicating private groups and is the idea that you never specify this group and that information is taken from tribes2? So it creates something like the template for you.

[arj03]

I wonder if it should also add invitations purpose to hops something :? So that the groups is also about people you are in a group with.

[staltz]

Yes, something like that. The keys 0, 1, 2 refer to hops from ssb-friends, and you can think of this as a (mathematical) set of peers. So the key group is also a set, it's all the peers that you share a group with. Those peers may overlap with some hops set, which means we'll have to do a union of the templates that match for that peer.

About the invitations feed, we have to consider whether it's okay for someone outside of your hops to invite you to a group. I think this shouldn't happen, only people within your hops can add you to a group. Because if anyone on the planet can invite you to a group, how will you know you have to replicate them to see the group invitation msg?

So this means we can just add { purpose: 'invitations' } in the template under hops 1 or hops 2. We can let implementations choose whether only friends can add you to a group, or if also friends of friends (hops 2) can invite, or beyond (hops 3).

[staltz]

One idea that came to my mind is whether we should have different tiers of groups, like "important" groups versus "casual" groups. Because maybe people who are in an important group, you'll want to replicate their groupFeed+abouts+contacts, but for people in the casual group you'll want to replicate groupFeed+abouts only.

But I realized there is no point in complicating things at this point. This can be a future feature.

[arj03]

no point in complicating things at this point

Yeah, lets keep things simple first.

So this means we can just add { purpose: 'invitations' } in the template under hops 1 or hops 2.

Agree!

[Powersource]

@arj03 andre and i were on a call and were confused about the best way of getting all the members of all your groups, live, from tribes2. The easiest way is to call list() and then listMembers() and do that on a setInterval lol but we wanna do it as a live stream. This is how the listMembers function looks right now.

https://github.com/ssbc/ssb-tribes2/blob/edbee5517df479ea850f69e335cabb0966d7137e/index.js#L184-L196

We can't just add live: true since then if we later get added to a new group, then add-member messages we've already traversed that were previously encrypted (now not) will be missed

[arj03]

@Powersource I was looking through the stack and isn't the problem that we don't have a way to get notifications on group adds here: https://gitlab.com/ahau/lib/ssb-keyring/-/blob/master/models/group.js#L59. If keyring had that, then we could use that here: https://github.com/ssbc/ssb-box2/blob/master/format.js#L77 and make that one live. Then we could listen to both members and lists and on new groups get all the members?

[staltz]

Or... perhaps tribes2 could have a live stream of all "newly discovered group secrets" after reindexEncrypted is done, this way we'll know we can repeat one of the isDecrypted('box2') queries to discover the members.

[staltz]

PR ready for review! @arj03 @Powersource

[arj03]

Had a shallow look through the code. It's hard to review the req-manager changes but overall things look good and the tests are really nice to see. It's coming together! We can leave the list thing for later, just as long as there is an issue for it. Because in production it's going to be pretty bad with the interval :)

[staltz]

Yeah I expect the live stream for member lists is going to be fixed soon, like this month, much before production.

About ReqManager, the biggest structural change is that the Maps don't have hops information anymore. That was actually a nightmare before, because hops num comes from ssb-friends, but then we were storing it in N places, and updating those dynamically as the algorithm switched from full to partial, or switched from following to blocking.

Now, there is only one hopsCache which is only updated when ssb-friends gives us new information.

On top of these changes, a couple other shuffling inside ReqManager that made it easier to check for group branches etc. And then, deletion of some redundant code.

[arj03]

Had a second look. The changes looks good, can't find anything except the minor nitpick for readability :) Given that existing tests pass and that the new group tests also pass I'll approve this.

[staltz]

Thanks for the review! Exciting to get this merged (and e.g. used in browser core)