search

spegel-org / spegel

Stateless cluster local OCI registry mirror.
MIT License
976 stars 47 forks source link

Evaluate future of libp2p-kad-dht #154

Open phillebaba opened 12 months ago

phillebaba commented 12 months ago

There is a new project https://github.com/plprobelab/go-kademlia that aims to build a standardized generic Kademlia implementation. It looks like libp2p2 kad will refactor to use the new library according to https://github.com/libp2p/go-libp2p-kad-dht/issues/856. I am currently not sure but I think we could just switch to using go-kademlia if it offers all of the features that we require. This may simplify some of the routing implementations as the new API is simpler.

dennis-tra commented 10 months ago

Hey @phillebaba, I just saw this issue here linked from https://github.com/libp2p/go-libp2p-kad-dht/issues/856. I'm one of the developers rewriting the DHT implementation for Kubo (IPFS).

I'd be curious to hear more about the features that you require. Right now, we're quite focused on the requirements that IPFS has, so your input could be very valuable here.

phillebaba commented 10 months ago

@dennis-tra I do not think it is a question of what features are needed on top of IPFS but rather which features Spegel does not require. At times I feel like libp2p KAD DHT has too many features making it difficult to understand the internals. The challenges that Spegel and IPFS has are obviously different as on needs to cater to public internet wide networks while Spegel runs within a private trusted cluster.

Having said that these are the two main issues that I am currently facing.

  1. Requiring a public key to be a part of the peer ID makes bootstrapping and restarts overly complicated. Currently the key is generated in runtime meaning that any restart of Spegel will result in a new peer ID. It also means that the bootstrap peer ID needs to be advertised through the Kubernetes API. It would be nice to not have to do this.

  2. I have received questions from end users about how locality works when it comes to peer priority. There are financial benefits from keeping traffic within the same zones on top of performance benefits. I have postponed giving any answer to this question as I do not understand the internals for libp2p KAD.

phillebaba commented 9 months ago

Looks like v2 of go-pibp2p-kad-dht will live under a different name.

https://github.com/plprobelab/zikade

This is a note to come back and check progress of the project at a later time. When stable it would be a good idea to migrate.