Testground epic

[y0sher]

Motivation

Our current CI

Utilizes K8S and python’s pytest, it runs ELK with fluentbit, collecting logs that are used for testing and for later for debugging. .It also runs a Pyroscope instance for viewing profiles. Same code also manages the deployment of bootstrap, poets, miners and wiring them together. The test setups the the configurations and interacts with the entities using the API. It then queries Elasticsearch or the API for data to run assertions otheron the test state.

Pros:

• Flexible, in-house system - we can do anything we want here.
• K8s allows easily adding new services (metrics, logging, other stuff) and scaling them
• Auto-scale-down saves money

  Cons:

• Slow heavy lifting to create the infrastructure for each test
• In-house bugs can make it hard to relay on test results
• Real-time testing using logs is prone to errors
• Hard to develop tests without expensive cloud-gear. (we never tried minikube with it..)

Testground

Testground is a platform for testing, benchmarking, and simulating distributed and p2p systems at scale. It's designed to be multi-lingual and runtime-agnostic, scaling gracefully from 2 to 10k instances, only when needed.

Pros

• Testground was used for testing IPFS, libp2p and Filecoin.
• whitebox testing! The tests are written in a way that allows access to the code that runs internally almost like a unit test.
• Built-in helpers like sync (syncing data between instances), network shaping and more.
• Supports anything that runs inside a docker
• Go-SDK
• same test works for different running environments from local binaries to local docker setup to a k8s cluster in the cloud, this allows developing locally on simple machines and getting rapid feedback.
• Written in go, @daonb already contributed a bug fix that got merged

Cons

• Still a bit immature
• need to re-write all tests
• need to move from using elk probably
• only supports aws for k8s
• might not be easy to change and adapt to our needs?

Running

Prerequisites: Go 1.16+, Docker daemon running.

$ git clone https://github.com/testground/testground.git

$ cd testground

$ make install       # builds testground and the Docker image, used by the local:docker runner.

$ testground daemon  # will start the daemon listening on localhost:8042 by default.

# => open a different console (client-side), in the same directory (testground/testground repo checkout)

# import the network test plan from this repo into $TESTGROUND_HOME/plans
# read more about this path at https://docs.testground.ai/getting-started#running-testground
$ testground plan import --from ./plans/network

# run two instances of the `start` test case from the `network` plan,
# building with docker:go, running with local:docker
$ testground run single --plan=network --testcase=start \
                        --builder=docker:generic --runner=local:docker \
                        --instances=5

TODO: (@y0sher): Add docs about reading logs and using node API and metrics. The test currently runs 1 PoET, 1 bootstrap ,3 miners and publishes the addresses of PoET and bootstrap.

(TODO: create issues and link)

Changes

• [x] - Run spacemesh network inside testground (#2589)
• [ ] - Provide metrics and log reading tools
• [ ] - provide test writing and running instructions, helpers (SM api client, metrics access, mesh and state access) + docs
• [ ] - write the first system test in testground (what should it be? should we start with basic p2p communication or go all in on mining test.
• [ ] - create as test plan and distribute test writing tasks to testing and core team
• [ ] - create an automated process to run test on changes in go-spacemesh using testground and collect results
• [ ] - run big scale tests on latest code nightly/weekly
• [ ] - start writing adversarial tests

[y0sher]

I'm raising here some concerns I have with testground, I'll edit this comment later and add more details

• Not very mature

  • Testground is on version 0.5.3
  • some features aren't fully supported :

    
    (🌕 = fully supported // 🌑 = planned)

  Experimental/iterative development 🌖 (The team at Protocol Labs has used Testground extensively to evaluate protocol changes in large networks, simulate attacks, measure algorithmic improvements across network boundaries, etc.)

  you can write tests that can scale pretty easily and test locally with not much instances - huge plus Debugging 🌗 - if you write metrics from the test plan code they are saved, there's no log aggregator or metrics collected outside the ones written manually. (these might be easy to solve though no idea whats testground planning to do in the future on that side)

  Comparative testing 🌖 Backwards/forward-compatibility testing 🌖 testground has some nice features for running tests with different versions of code and checking backward compatibility and such.

  Interoperability testing 🌑 Continuous integration 🌑 CI integration needed from scratch probably. //todo check with the testground team their plans for that.

  Stakeholder/acceptance testing 🌑

• Building C-GO libraries means we are downloading everything everytime and it takes longer than we planned. takes about 3 minutes to start 5 miners locally. need to work on bigger tests to get real numbers.