search

cosmos / cosmos-sdk

:chains: A Framework for Building High Value Public Blockchains :sparkles:
https://cosmos.network/
Apache License 2.0
6.17k stars 3.57k forks source link

[Feature]: modular testutil.Network #18145

Closed tac0turtle closed 10 months ago

tac0turtle commented 10 months ago

Summary

Testutil.Network is being used to spin up n number of validators using CometBFT. The Cosmos SDK and its users use this package in order to test their applications.

We have seen more testing frameworks come up for different needs, levels of testing and for different consensus

We should swap testutil.Network to an interface that can be used to against many different environments.

Problem Definition

Streamline testing with different environments without needing to rewrite tests

Proposed Feature

Modify testutil.Network into an interface that testing framework authors can implement so that e2e tests can be run against many different environments.

alexanderbez commented 10 months ago

Specifically, I'd like to see us use more practical environments to test RCs and regression test against prior releases. Most likely Starship will help us a ton here.

Anmol1696 commented 10 months ago

@alexanderbez that is something we are working on. Here are some more details. If there is access to a development kubernetes cluster, (which we could hook up with gh-actions) then we should be able to create a large scale multi-node environment, and depending on resource consumption (which we can optimize for), should be able to run regression tests parallely as well.

I am working towards full monitoring support (so we could also generate reports), and also perform benchmarking and performance reports as well, and be able to compare releases in a much more realistic and automated fashion.

The one thing i am not super sure about, is what exactly to be running in the regression tests. This is where reusing the testutil.Network and being able to run e2e tests (even a subset) against multiple backends (including Starship) will help alot.

Some of limitations of an e2e system that runs everything as a blackbox, is lack of access directly to keepers and app internals. This makes it slightly more tricky around testing weird edge cases, but it would really make sense for regression tests. If we can abstract out all the special calls to the network in the testutil.Network interface, then we could implment the handlers for Starship, to be able to reuse e2e tests for regression testing as well. This could even include some admin functions (which we could run partially with cometmock).

robert-zaremba commented 9 months ago

I see that task has been closed. What is the decision and the future direction?

tac0turtle commented 9 months ago

we merged the modularity and starship is working on integrating the interface