search

Drops-of-Diamond / diamond_drops

WIP on sharding and Ethereum 2.0 with enshrined-in-consensus data availability and Rust: a fast, safe, concurrent and practical programming language
The Unlicense
56 stars 14 forks source link

P2P sharding network(s) #34

Open jamesray1 opened 6 years ago

jamesray1 commented 6 years ago

Have the ability of building and sending collations across those networks.

https://github.com/libp2p/specs

https://github.com/libp2p/rust-libp2p (this is a WIP), but devp2p is not maintained and is poorly documented, so it's still our best option.

jamesray1 commented 6 years ago

Or have one network...

https://ethresear.ch/t/torus-shaped-sharding-network/1720/6

ltfschoen commented 6 years ago

Please review the following questions that I've prepared after reading through the libp2p spec at https://github.com/libp2p/specs. Try to challenge/answer the questions collectively as a team:

1) What libp2p protocol categories/roles and associated properties, security models, and respective network size are we using on each layer of our stack, and what are scenarios where multiple protocols would run simultaneously?

2) What collection of transport protocols will we be running libp2p protocol over?

3) i) What port(s) will we use on the libp2p network layer to perform operations for each of the transport protocols that we’ll be using? ii) What is an example scenario where we’ll be protocol multiplexing multiple transport protocols (i.e. UTP, UDT, WebSockets, SCTP, WebRTC, TCP Remy) through a single port? iii) What is an example scenarios where we'll be multiplexing reliable stream(s) or unreliable datagram(s) in a connection from an IPFS node to other node(s) on the a transport protocol in the network layer of the communication model? iv) What protocol(s) would we be mounting on top of libp2p with the connection? v) What type of connection would we be using? vi) What platforms and network setups will we need to libp2p to support? vii) How will we build a multiaddr-based network package module like go-multiaddr-net in Rust, for IPFS to delegate the transport dialing to, and for scoping the implementation of other transport protocols. viii) What portable format encodings do we need to use for protocol messages (i.e. protobuf)? ix) What are scenarios when we will use multistream, multistream-select, and Stream Multiplexing? What default multiplexor will we use with Stream Multiplexing (i.e. HTTP/2, SPDY, QUIC, or SSH)? x) How will we use Multicodecs for each IPFS Protocol that we use?

4) i) How do we need to encrypt communications on libp2p for security in our implementation? Under what scenarios would users not need to operate with encrypted communications? ii) Is libp2p's application of cyphers with only a minimal portion of the whole TLS standard used sufficient for our implementation?

5) i) How will users running a node behind a NAT traverse it in order to establish connections to peers in our implementation of libp2p's IPFS? ii) Will we use libp2p's ICE-like protocol for full NAT traversal or will we use a different NAT traversal library instead of libnice, such as libwebrtc or natty? iii) How will we check if our implementation of NAT traversal is interoperable? iv) How will we implement optional fallback to connection relaying communication as a transport that may be turned off by users to ensure a full connectivity graph?

6) What network topologies will we be using for our system's routing mechanisms, peer discovery, and message propagation?

7) i) What information will records (IPFS Linked Data) in our network store for discoverability? ii) How will we use libp2p to locate resources in the network and side channels? Reference: https://github.com/libp2p/specs/blob/master/3-requirements.md#37-resource-discovery

8) What libp2p messaging protocols will we let users use after they open a new stream and register a stream handler and why?

9) How will we adopt naming conventions in our implementation of the network?

10) What is a scenario where different libp2p subsystem interfaces (i.e. Peer Routing, Swarm, Distributed Record Store, and Discovery) would use each other in our system implementation? Reference: https://github.com/libp2p/specs/blob/master/4-architecture.md#4-architecture

11) What Peer Routing subsystem mechanism will we implement and why? Reference: https://github.com/libp2p/specs/blob/master/4-architecture.md#41-peer-routing

12) What are different Swarm scenarios that we need to handle (where we are using a stream)?

13) What Distributed Record Store system will we use and why?

14) i) What Discovery Protocols will we use and for what scenarios? ii) Will our Discovery Protocol need encrypted discovery beacons (i.e. mDNS)? iii) Would we use a Bootstrap-List Discovery Protocol and make it use configurable?

15) Which of the libp2p modules (transports) listed at the following reference link will we be using with the Rust implementation of libp2p https://github.com/libp2p/rust-libp2p?

16) How will we use the Swarm Dialer?

jamesray1 commented 6 years ago

I don't think we'll use Swarm as data will be stored on the blockchain, but it could also be done with Swarm as a complementary storage service.

i) How will users running a node behind a NAT traverse it in order to establish connections to peers in our implementation of libp2p's IPFS?

For discovery mDNS-discovery looks like a low priority since it's designed for LANs. random-walk seems like it may be a good candidate. Due to the partial trust with bootstrap list, that is another low priority / enhancement alternative.

For relay when it is not feasible to reach a peer through a NAT, https://github.com/libp2p/specs/tree/master/relay would be useful, however rust-libp2p is unfortunately not listed as an implementor of it.

jamesray1 commented 6 years ago

https://github.com/libp2p/rust-libp2p/issues/187

jamesray1 commented 6 years ago

https://github.com/libp2p/go-floodsub/issues/77

https://ethresear.ch/t/topology-on-libp2p-libp2p-design/2028/3?u=jamesray1

WIP on answering the above questions, thanks for asking.

jamesray1 commented 6 years ago

rustlib-p2p has implemented relay-0.1.0: https://github.com/libp2p/rust-libp2p/tree/master/relay.

jamesray1 commented 6 years ago

I'll probably implement gossipsub since it is preferable to floodsub.

jamesray1 commented 6 years ago

https://github.com/Drops-of-Diamond/diamond_drops/issues/94

jamesray1 commented 6 years ago

"Exploring Ethereum on libp2p" https://github.com/libp2p/libp2p/issues/33#issuecomment-390508321

jamesray1 commented 6 years ago

I just want to clarify that we don't need to design a libp2p implementation from scratch, a lot of work has already been done with rust-libp2p, plus other Rust modules. I've created a wiki page with your questions here, and will try to answer them in due course.

jamesray1 commented 6 years ago

I've answered about half of the questions, might come back to them some time later after writing gossipsub and looking at other priorities.

ghost commented 6 years ago

@jamesray1 hey, I'll be joining the meetings in Berlin later this week for the libp2p team and starting to prep for it -- is the current state of the questions in the comment above or on the wiki page you mentioned?

jamesray1 commented 6 years ago

@lgierth hi, it's on the wiki page. I haven't bothered to answer all of them yet as I was tending to repeat myself by referring back to rust-libp2p, which Luke didn't seem to be aware of.

Right now I'm reading through tokio (a Rust crate for async networking), and before that was reading up on libp2p, rust-libp2p, Go, gossipsub, etc. Will start developing gossipsub soon.