feat: persistent kad store

[ramfox]

right now our kademlia store is mem only. Which means each time the p2p node is restarted, we are no longer providing any records.

We need a persistent store and to extend the current p2p config so we can configure the kademlia store.

[Winterhuman]

Just wanted to add that https://github.com/ipfs/kubo/issues/3926 might be worth considering when storing Peer Records. Accounting for storing long-lasting peers as future bootstrap nodes would be a nice addition (please, say if you'd rather I open a separate issue for this)

EDIT: I've opened an issue at rust-libp2p instead since it mirrors the effort to persist the routing table in go-libp2p, rather than in Kubo: https://github.com/libp2p/rust-libp2p/issues/2987

[rklaehn]

@Winterhuman I think whatever you can do to make the bootstrap process more resilient would be great. The current state where you have a number of hardcoded bootstrap nodes is not really great.

It would e.g. mean that a region that is partitioned from the rest of the world would just stop working because nodes can't find each other anymore. Very bad. E.g. Russia starts a "big firewall" and all hardcoded bootstrap nodes are outside.

However, I think this might be best done in a separate issue.

[dignifiedquire]

Bootstrapping based on existing nodes is great, but needs considerable security evaluation, as just connecting to those opens you easily up to eclipse attacks.

[rklaehn]

OK, so I think this is two parts. I looked at the storage API. One part is remembering what we ourselves provide. That is so closely related to the store that it might be best to do it in the metadata table of the store or something. We just need to remember which cids we have we want to also provide.

    /// Gets a record from the store, given its key.
    fn get(&'a self, k: &Key) -> Option<Cow<'_, Record>>;

    /// Puts a record into the store.
    fn put(&'a mut self, r: Record) -> Result<()>;

    /// Removes the record with the given key from the store.
    fn remove(&'a mut self, k: &Key);

    /// Gets an iterator over all (value-) records currently stored.
    fn records(&'a self) -> Self::RecordsIter;

The other is to persist provider records that other peers have given us. That is more about being a good p2p citizen. I think it would suck from an ops pov to have multiple dbs, but this could be an entirely separate table in our db.

    /// Adds a provider record to the store.
    ///
    /// A record store only needs to store a number of provider records
    /// for a key corresponding to the replication factor and should
    /// store those records whose providers are closest to the key.
    fn add_provider(&'a mut self, record: ProviderRecord) -> Result<()>;

    /// Gets a copy of the stored provider records for the given key.
    fn providers(&'a self, key: &Key) -> Vec<ProviderRecord>;

    /// Gets an iterator over all stored provider records for which the
    /// node owning the store is itself the provider.
    fn provided(&'a self) -> Self::ProvidedIter;

    /// Removes a provider record from the store.
    fn remove_provider(&'a mut self, k: &Key, p: &PeerId);

[rklaehn]

So the bad news is that getting this done by 0.1.0 is hopeless unless we postpone the release. The good news is that I am quite confident that the in memory record store (the libp2p kad default implementation) won't be the limiting factor for 0.1.0. It can easily handle ~100k records, which corresponds to ~20G of unixfs v1 data at maximum chunk size.

The reason is that while the rad record store is a bit weird, they do not store addresses for local records. So one local record takes up relatively little space in the store.

The biggest problem then will be all the traffic from reproviding all these records, but that has nothing to do with whether the kad store is persistent or not.

So I would propose to just have code that adds all cids currently in the store to the in memory RecordStore on p2p startup, and call it a day for 0.1.0.

See https://github.com/n0-computer/beetle/issues/140