Staged Test Network Plan

[pipermerriam]

I would like to outline some proposed structure for how we roll out our test network.

The goal of this test network is to iterate towards the full functionality necessary for the Chain History network.

Phases

Phase Zero: Overlay Only

During this phase we will simply establish our ability to establish a stable overlay network using the Portal Wire Protocol messages.

• A: Clients support can be externally configured in some way to specify bootnodes.
• B: Clients can use use the Portal Wire Protocol using the protocol identifier: 0x500B (History Network)
• C: Clients can send and receive PING/PONG messages using the overlay protocol
  • "fake" radius information in the payload
• D: Clients can send and respond to FINDNODE/FOUNDNODES messages using the overlay protocol.
• E: Clients periodically check the liveliness of nodes in their routing table and evict unreachable nodes
• F: Clients actively populate their routing table by exploring the DHT (typically via random exploration using the recursive find nodes algorithm)
• G: Client has published instructions on how to build, configure, and launch a node for the test network.

In this phase, each team will be responsible for deploying however many nodes they wish. Each team may supply the ENR records for any stable "bootnodes" they will be operating.

What we are testing:

1. Client interoperability and stability
   • Message encoding and decoding for PING/PONG/FINDNODE/FOUNDNODES
2. Healthy overlay network
   • Do clients populate and maintain their routing tables as expected.
   • Can we navigate the network as expected to find nodes the closest nodes to a location.

Client	A	B	C	D	E	F	G
Fluffy	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
Trin	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
Ultralight	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:

Phase One: Simple Content Retrieval

During this phase we will be testing the ability to find and retrieve content stored in the network by other nodes.

• A. Clients have a basic implementation of a content database
  • The database can be manually populated with data
  • support for looking up a piece of content by it's content-id
• B. Clients have support for the content-key format for headers referenced by their block hash.
• C. Clients support a subset of the functionality for serving FINDCONTENT requests
  • Clients only need to support responding when the payload can be returned within the CONTENT message response.
  • Clients do not need to support uTP based transmission at this stage
• D. A client can be launched with a pre-populated database of content
  • In this phase we will populate our test network with nodes which have had their content databases pre-populated.
  • We will likely do something like pre-loading the first 1 million block headers

What we are testing

1. The ability for nodes to traverse the DHT and find the nodes which have the content they need
   • Any block header in the first million blocks should be retrievable by its block hash.
2. The basic transmission of content without uTP

Client	A	B	C	D
Fluffy	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
Trin	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
Ultralight	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:

Phase Two: Gossip

During this phase we will build out the mechanisms needed for gossip, including use of the uTP protocol.

• A. Clients support uTP
  • Listening for inbound connections
  • Establishing outbound connections
  • Receiving data sent over the uTP stream
• B. Client support for gossip
  • Responding to OFFER messages if the content is of interest
  • Establishing outbound connection upon receiving an ACCEPT message and sending the data over uTP
  • Listening for an inbound connection after sending an ACCEPT message and receiving the data over uTP
  • Support for the block header content-key and validation of the block header.
  • Upon successful receipt of new content, performing neighborhood gossip.
• C. Client support for the POKE mechanic.
  • Tracking of the nodes along the search path which did not have the content but should have been interested.
  • Once content was successfully retrieved, offering it to nodes along the search path.
• D. Non critical but related functionality
  1. Management of the content database, eviction when full based on radius
  2. Validation that offered headers are indeed part of the canonical chain (depends on header accumulator)

What we are testing

1. uTP based transmission of content
2. The effective spread of new content using gossip
3. The passive replication of existing content using POKE mechanic.

Client	A	B	C	D.i	D.ii
Fluffy	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:pizza:
Trin	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:pizza:
Ultralight	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:question:	:heavy_check_mark:

Phase Three: Head Tracking

TODO: we can no longer simply follow the chain with the highest TDD after merge. Need to implement light protocol for Eth2

NOTE: establish the "correct" solution and then look for a quick solution that will get us the needed functionality quickly.

Phase Four: Full Content Transmission

During this phase we will flesh out the remaining functionality for transmission of content, specifically, large payloads that must be sent over uTP

• A. Clients support CONTENT response that contain a uTP connection ID
  • Upon receipt of a CONTENT response with a uTP connection-id, the client initiates a uTP stream and receives the content payload over the stream.
• B. Clients serve FINDCONTENT requests for all content types (headers, bodies, receipt bundles, master and epoch accumulators)
• C. Clients support block body and receipt bundle content types
• D. Received block body and receipt content is validated using the block header.

What we are testing

1. support for remaining content types (bodies and receipt bundles)
2. clients implement full support for FINDCONTENT/CONTENT including full uTP based content transmission.

Client	A	B	C	D
Fluffy	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:
Trin	:heavy_check_mark:	:pizza:	:heavy_check_mark:	:heavy_check_mark:
Ultralight	:heavy_check_mark:	:heavy_check_mark:	:pizza:	:heavy_check_mark:

Phase Five: The Road To Production

TODO: what is left?

NOTE: visibility into the state of the network is probably valuable at this stage.

NOTE: verifying cross client compatibility will need to be done

[pipermerriam]

Proposal for DevConnect goals:

1. Each client has documentation on how to install and launch a node which will connect to the current testnet bootnodes.
2. Ultralight has some documentation on how to launch and run the in-browser stuff.
3. Hosting of at least a subset of the header chain available via the testnet.

[ogenev]

3. Hosting of at least a subset of the header chain available via the testnet.

Does this include 1) and 2) from Phase Two: Gossip?

[pipermerriam]

Does this include 1) and 2) from Phase Two: Gossip?

I think it can be done in whatever the fastest/quickest way that works is which doesn't diverge too far from the existing roadmap. Don't want us doing extra work that doesn't lead us towards the final product... but if we can have uTP streams by then in any capacity then huzzah!

[acolytec3]

Have updated for Ultralight

[kdeme]

As have I for Fluffy

[ogenev]

Updated for Trin

[pipermerriam]

is :pizza: a no? partial?

[acolytec3]

is :pizza: a no? partial?

Partial in our case

[ogenev]

is pizza a no? partial?

Partial/in progress for Trin. Updated POKE with :x: as It is not yet implemented.

[carver]

POKE in trin is merged, I updated the table

[kdeme]

Closing as this is completed since a while for all clients mentioned in the issue tracker.