Standardize testnet and testing infrastructure

y0sher commented 3 years ago

Motivation

We have about 5 (probably) more tools to create a network of miners whether on some cloud provider or locally or using k8s. Each of these tools has a set of important features, while lacking another set of important features.

Currently we have different tools used for the same task on different contexts, for example different code is orchestrating starting a testnet (go-spacecraft) than used to create a network for the CI system tests (go-spacemesh/tests). We will forever need to maintain multiple systems with essentially similar features and hunt down differences..

What do we need

We want to stop creating new tools and direct our focus to one (or more than one if they'll be modular) infrastructures that will be able to:

Setup a network (do the bootstrap and poet startup seq)
control network (run code on miners, update versions, kill, add)
monitor network (logs, metrics, machine data, profiling, tracing, basic network state like miners amount)
Run certain scenario on a network (basically a test, send tx and wait for it to be processed for example, but eventually much more complex tests like bad nodes) NOTE: this relies on some features I've mentioned above.
Generate reports about network health, errors, flow etc, infra health..
feature we need to discuss: Testing the infra to rule out protocol failures faster.

if we will have all our team on this one project we'll be able to deliver features and bug fixes faster and on a single place.

Next steps

As I see it now, we have a few limited options if ruling our using an external existing testing infra (which @daonb is looking at) or writing a whole new thing again which we now going to take us back to the same place eventually.

We need to decide either somehow run our existing tests on go-spacecraft deployed infra or break the network deployment apart from the CI and use it for running testnets as well. Also there are some inevitable changes we'll need to go through to cover up for the missing features in any path we chose.

Please share your thoughts- @daonb @narayanprusty @AmitShaul or anyone else

y0sher commented 3 years ago

From @avive:

I'd like to chime in from a product perspective. I think that it is important for us to be crystal-clear about the requirements from the infra by looking at the 3 main main use-cases we have for a spacemesh network, which are:
Infra for running CI / tests.
Infra for open source devs to be able to spin up a local dev net for go-sm dev purposes (aka localnoet).
Infra for long-running networks (aka testnets).
Each use-case has different requirements and network expected life-span. I think that the requirements you mentioned above are only applicable for some use cases but not other. So, I think it will be quite useful to define the requirements for each use case separately first, and not try to have a union list of all requirements from all uses cases because you guys might decide, for example, that localNet is best maintained and updated by the current localnet repo scripts and not by migrating it to go-spacecraft based on its requirements... Hope this makes sense...

y0sher commented 3 years ago

From @avive:
I'd like to chime in from a product perspective. I think that it is important for us to be crystal-clear about the requirements from the infra by looking at the 3 main main use-cases we have for a spacemesh network, which are:
Infra for running CI / tests.
Infra for open source devs to be able to spin up a local dev net for go-sm dev purposes (aka localnoet).
Infra for long-running networks (aka testnets).
Each use-case has different requirements and network expected life-span. I think that the requirements you mentioned above are only applicable for some use cases but not other. So, I think it will be quite useful to define the requirements for each use case separately first, and not try to have a union list of all requirements from all uses cases because you guys might decide, for example, that localNet is best maintained and updated by the current localnet repo scripts and not by migrating it to go-spacecraft based on its requirements... Hope this makes sense...

It makes sense to keep localNet alive for the open source devs and running without cloud infra, though it will still be awesome if it could run the tests we have in the ci and let developers write their own tests or just play with their network. this is what I'm trying to achieve, if these tools would share a lot of logic it might be possible.

daonb commented 3 years ago

Great summary. I feel the pain of many half-baked tools and I don't believe we should develop another one. I'd rather improve one of the tools or adopt an open source tool that fits.

I like the idea of narrowing it down to one tool but it comes with the risk of having a bloated. It doesn't have to be - gcloud is proof that one tool can do much more than what we need. We just need to design the CLI before we start coding and do it for the "dream" version and not for the first.

lrettig commented 1 year ago

This would still be useful - it's still too hard to run a local testnet

spacemeshos / pm