Recon

[AaronGoldman]

A proposal synchronization of ranges of EventIDs from streams.

[oed]

Please use cip-124

[oed]

📚 Preview

[oed]

It might be nice to be able to reference a range as a ceramic://... uri

[qbig]

@AaronGoldman @oed would using a bloom filter in addition to a hash during set recon make it more efficient (less round trips)? https://www.youtube.com/watch?v=xuddEiu-t-8&ab_channel=HeidiHoward

you:  [ your initial keys ]
they: [ their initial keys ]
    -> ( request ) [ their keys after processing request]
    <- ( response ) [ your keys after processing response]

Current

you:  [ape,eel,fox,gnu]
they: [bee,cat,doe,eel,fox,hog]
    -> (ape, h(eel,fox), gnu) [ape,bee,cat,doe,eel,fox,gnu,hog]
    <- (ape, h(bee,cat), doe, h(eel,fox), gnu, 0, hog) [ape,doe,eel,fox,gnu,hog]
    -> (ape, 0, doe, h(eel,fox,gnu), hog) [ape,bee,cat,doe,eel,fox,gnu,hog]
    <- (ape, 0, bee, 0, cat, h(doe,eel,fox,gnu), hog) [ape,bee,cat,doe,eel,fox,gnu,hog]
    -> (ape, h(bee,cat,doe,eel,fox,gnu), hog) [ape,bee,cat,doe,eel,fox,gnu,hog]
    <- (ape, h(bee,cat,doe,eel,fox,gnu), hog) [ape,bee,cat,doe,eel,fox,gnu,hog]

After using a bloom filter

you:  [ape,eel,fox,gnu]
they: [bee,cat,doe,eel,fox,hog]
    -> ([ape, h(eel,fox), gnu] + bloom(eel,fox)) [ape,bee,cat,doe,eel,fox,gnu,hog]
    <- ([bee,cat,doe,hog] + [ape, h(bee,cat), doe, h(eel,fox), gnu, 0, hog] + bloom(ape..hog)) [ape,bee,cat,doe,eel,fox,gnu,hog]

[AaronGoldman]

fission's car-mirror work a little like that. https://talk.fission.codes/t/notes-through-the-multiverse-ipfs-car-pool-car-mirror/3647

Certainly the split range function could avoid sending keys in the bloom filter as the splits. This would reduce the likelihood of sending duplicate data at the cost of the bloom filter bandwidth. Particularly as you get close to the leaves where there are not huge numbers of keys in the ranges it is likely to save the last few rounds.

https://hur.st/bloomfilter/?n=&p=.001&m=2KiB&k=

e.g. If we through in a 2KiB filter we could probably send all remaining key when we got to about 1,000 keys in a range. Speeding up the last few rounds.

[oed]

@qbig @AaronGoldman interesting discussion. Can we continue in the discussion thread for this CIP? https://forum.ceramic.network/t/cip-124-recon-tip-synchronization-protocol/1144