EPIC: interop testing for go and rust libp2p - Ping Test

[laurentsenta]

We want to run a "simple" test (ping) that verifies interoperability between versions and implementations. This test should run at least before every (go|rust) libp2p release.

Use case:

• Test cross-version compatibility for Go
• Test cross-version compatibility for Rust
• Test cross-implementation compatibility for Rust and Go

We're aiming for:

• a single ping test implemented in different languages (go + rust)

• running interop + cross version for every PR on the go-libp2p repo

• running interop + cross version for every PR on the rust-libp2p repo

• @mxinden and @marten-seemann agreed to run this test if

  • it takes < 30 minutes
  • it's stable (aiming for < 1% failure rate)

• they want "reasonable" run time in the future

  • my goal right now is < 15 minutes which is the go test runtime on ubuntu,
  • I'm trying to push it down to ~ 5 minutes with caching. There is a O(n) relationship between build time & the number of tested versions and caching would make it O(1).
  • we're going to timebox this effort for a week, then discuss how far we want to go.

Todo

• [x] implement cross-version interop testing in go
  • https://github.com/testground/testground/issues/1340
  • https://github.com/libp2p/test-plans/pull/23
• [x] implement cross-version interop testing in rust
  • https://github.com/libp2p/test-plans/pull/26
• [x] implement cross-implementation interop testing
  • https://github.com/libp2p/test-plans/pull/26
• [x] maintain & enables tests in go and rust repos
  • https://github.com/libp2p/rust-libp2p/pull/2835
  • https://github.com/libp2p/go-libp2p/pull/1725
• [ ] stability & perf improvements
  • https://github.com/libp2p/test-plans/pull/34
  • https://github.com/testground/testground/pull/1407
  • https://github.com/testground/testground/issues/1401
  • https://github.com/libp2p/test-plans/pull/32
  • https://github.com/libp2p/test-plans/issues/31
• [ ] Sync with the libp2p/interop repo https://github.com/libp2p/interop/issues

References:

• https://www.notion.so/pl-strflt/Roadmap-Draft-c6b7a9e67b18468bac16cccb9a8bf8fe
• https://www.notion.so/pl-strflt/Testground-Interop-Features-Plan-with-Stability-in-Mind-dad6b6d25b79451ba01e0a28da1888b3

[laurentsenta]

Gathering content in a single issue to share w/Bloxico.

[laurentsenta]

Follow-Up discussion

• Can we get rid of the building step when we use a test from another implementation:
  • We need to make sure an update in a repo doesn't break another, it's very important for stability during do cross-implementation interop tests. Fixes like the one in https://github.com/libp2p/test-plans/pull/45 might be working, but it's hard to make sure we're covering all cases where a build might fail.
  • We don't want to "pin forever" the test artifacts either because a test specification might change (for example, how we sync instances).
  • What if we built every artifact for every known version on merges to libp2p/test-plans?
  • Then reuse these artifacts during test runs?
  • This way implementation langX-libp2p never attempt to build a test for langY-libp2p.
• Another important point is how to deal with new feature:
  • "as a langX-libp2p maintainer, I want to create a test in this plan than will be used by langY-libp2p"
  • It might be a feature that only exists on master,
  • It might be a feature that doesn't exist in previous versions (we're only testing ping at the moment).