Utp cross client testing

[KonradStaniec]

As it currently stands there are 3 upcoming utp implementations : fluffy, trin and ultralight. It would be great to establish few practices and test cases which would guarantee compatibility between all of the clients.

Things we can do (in increasing complexity order):

1. Generate test vectors for encoding and decoding of each utp packet (DATA, FIN, STATE, SYN, RESET), each client could use them in their implementation unit test suites.
2. Describe few test cases each implementation need to pass in theirs integration suites between 2 nodes of the same type.
3. Each implementation could create test app which would start uTP instance with some config and expose simple api (via command line , json-rpc or whatever we decide) like:

   connect
   read
   write
   close

   Having such app would make it possible to create CI integration test suite in each client or event test manually certain conditions between clients. Ideally we would also find/implement something similar to https://github.com/Shopify/toxiproxy but for udp transport (as discoverv5 works over udp) to inject network failures (lost packets, timeouts) while working on localhost

4. Imo gold standard test would be to have some small testnet (like 3 nodes, one of each type), with some monitoring attached, and every node would disseminate some content via OFFER/ACCEPT messages using utp. But this is more of the integration between utp subprotocol and history (or state) subprotocol.

It would be great to hear other teams thoughts on this topic :) cc: @victor-wei126 @acolytec3

[acolytec3]

Generate test vectors for encoding and decoding of each utp packet (DATA, FIN, STATE, SYN, RESET), each client could use them in their implementation unit test suites.

This should be doable for us in short order.

Describe few test cases each implementation need to pass in theirs integration suites between 2 nodes of the same type.

This would require us to have an integration test suite :grimacing: This is a bit farther off but still doable.

Each implementation could create test app which would start uTP instance with some config and expose simple api (via command line , json-rpc or whatever we decide) like:

We more or less already have this via our browser client. Should be easy enough to add whatever additional bits are needed to configure the stream, basically just agreed upon content to stream.

[oslfmt]

What is the exact difference between (2) and (3)?

Generate test vectors for encoding and decoding of each utp packet (DATA, FIN, STATE, SYN, RESET), each client could use them in their implementation unit test suites.

This is also doable for us. I think someone should create a test vector spec so we can test cross client compatibility.

Describe few test cases each implementation need to pass in theirs integration suites between 2 nodes of the same type.

By this I am thinking of tests like when A sends DATA B responds with STATE, and we test other things like handshake, etc. I suppose that can be done with ethportal_peertest--there is work being done to just spin up 2 trin nodes and pass tests between them. We could just start a utp connection and then pass messages, verifying the responses. @acolytec3 by "integration test suite" is what I just described somewhat to what you mean?

Each implementation could create test app which would start uTP instance with some config and expose simple api (via command line , json-rpc or whatever we decide)

It seems like ethportal_peertest could be used exactly for this. I feel like a good route to go would be doing (1) and then (3), since ultralight has browser client and trin has ethportal_peertest.

[acolytec3]

By this I am thinking of tests like when A sends DATA B responds with STATE, and we test other things like handshake, etc. I suppose that can be done with ethportal_peertest--there is work being done to just spin up 2 trin nodes and pass tests between them. We could just start a utp connection and then pass messages, verifying the responses. @acolytec3 by "integration test suite" is what I just described somewhat to what you mean?

More or less. It should be a lot easier for us now that I've got a functioning CLI version of our client available to work with that is using all the same existing code as the browser client. We can also support testing via ethportal_peertest as long as it is going to leverage json-rpc for the tests and just treat the portal network clients as a black box. We're going to be adding a simple jsonrpc interface for the CLI version of Ultralight in the next few weeks so exposing specific endpoints for testing should be trivial at that point.

[KonradStaniec]

what are capabilites/interface of this ethportal_peertest app ?

I think one question here do want to test uTP by itself or only with OFFER/ACCEPT (or both) ?

Imo the next step would be to create spec describing json-rpc interface that such uTP test app should implement. After that we could define required testcases.

[oslfmt]

It's been awhile since I've played with it, but last I checked ethportal_peertest is just a testing framework that spins up two trin nodes, one being the boot node and the other the test node, ie, one under test. Then the boot node issues a suite of tests, like sending ping, or findcontent requests, and the test node responds and we check if the expected response was received.

I think we should test uTP by itself first to make sure all implementations work and they work together. After that we can just treat it as a black box and it will be pretty easy to do full tests with OFFER/ACCEPT as those are just protocol messages.

[KonradStaniec]

I think we should test uTP by itself first to make sure all implementations work and they work together.

Totally agree !

So my proposal is: let each team build a simple test app with json rpc interface which mirrors common utp operations

This will enable following things (in order of complexity):

1. Manual testing of compatibility between implementations - everyone will have simple way of checking some scenarios
2. Building some test scripts which we can share between teams, to test some basic scenarios.
3. Build something similar to https://github.com/marten-seemann/quic-interop-runner, where each team could publish its test app in docket container, and we will be to inject some faults between app by ns3 network simulator.

I have implemented simple test app in nim - https://github.com/status-im/nim-eth/pull/461/files, just for starters.

Wdyt @acolytec3 @victor-wei126 ? If you are 👍 on this, I will create doc utp-test-app-spec.md which will describe json rpc apic this app should implement. Imo it should be as simple as possible, so the creating such app is not a big burden for each dev team.

[acolytec3]

My instincts say that most of this scope can already be covered by just directly triggering the OFFER/ACCEPT flow between the clients which Ultralight can already do without any additional work. If we want to go down this route, w could expose the specified RPC methods from our existing NodeJS Ultralight client so it's probably not a big deal from my perspective but maybe a little bit duplicative. My main concern is exactly the point you raised that it not be too complex an operation to build it independently of our main effort.

[KonradStaniec]

@acolytec3 on one hand I kinda agree with you that some of the scenarios can be triggered through using OFFER/ACCEPT flow, but on the other hand OFFER/ACCEPT add several failure modes on top of uTP and uTP itself is not so trivial protocol to get right.

My instinct here say that even if we won't need such app immediately (as we will be able to test some scenarios thorugh offer/accept) , ultimately we will hit some edge case scenarios which will force us to implement such app to test utp in isolation.

As I say this does not need to be anything fancy, only mirror utp interface (connect/write/read/close) through some external interface (imo json-rpc is good candidate here as every client in ethereum space have access to some library taking car of it)

[acolytec3]

Makes sense. Should be trivial for us to implement so I'll add that to our json-rpc this week.

[acolytec3]

Is there still an appetite to build out this separate uTP app for interop testing or are we satisfied that all of this is being covered by the existing uTP interop testing already done?