search

celestiaorg / celestia-app

Celestia consensus node
https://celestiaorg.github.io/celestia-app/
Apache License 2.0
345 stars 285 forks source link

EPIC: Post feature freeze testing #1256

Closed evan-forbes closed 8 months ago

evan-forbes commented 1 year ago

We have been pushing many breaking changes lately to fit them in before feature freeze at the end of the month. After feature freeze, we will want to spend more time building out robust tests for mainnet. This includes but is not limited to:

Stretch

MSevey commented 1 year ago

Something that might be good to add to the stretch goals is thinking about a framework for testing version upgrades and compatibility.

This could start by just ensuring we have the ability to launch nodes on different versions, and upgrade or downgrade a node without restarting the test.

evan-forbes commented 1 year ago

yes! good idea @MSevey, I totally forgot about #313, added to stretch goals

MSevey commented 1 year ago

nice yea, honestly i think that test #313 might make sense to bump up to a required and tie it into release. Basically launch the tests with half the nodes on the current path version and half on the new patch version as a sanity check that nothing breaks.

Then a future test could be more involved in terms of more patch versions, minor versions, and major versions to really stress test the network and understand compatibility issues. This one would of course be more of a debugging test. There might be some specific upgrade scenarios that we would want to build into the pipeline, but this and those would definitely be stretchy stretch goals.

Bidon15 commented 1 year ago

Thanks for the detail overview of the epic. It's Epic 😄 I would appreciate if we take our time to discuss how we approach this epic both from app and devops/testing teams

We can break down for example #1258 or compact blocks https://github.com/celestiaorg/celestia-core/issues/883 to decide who does what so at the end the tests bring the most value and insights to core/app team As well as we can learn how we can collaborate better for future joint issues

cmwaters commented 1 year ago

I think an overarching goal around our testing should be to put together some written artefact across the teams which can really act as a handbook making it clear for any team member or for anyone interested what we test, how we test it, when we test things and so forth. There is no monolithic one solution fits all kind of approach to testing. It really requires a myriad of different solutions that juggle the tradeoffs between different forms of coverage, "introspectability", time and resource costs. Thinking about testing the same systematic way we think about protocols and writing "specifications" will produce more reliable software. Having a place where people can read about everything will make it easier for us to cover our blindspots.

cmwaters commented 1 year ago

I also think @MSevey is totally spot on with his points. We should already be getting into the rhythm with upgrade and compatibility testing so it's like clockwork for us engineers by the time we hit our first mainnet upgrade. Previously at Tendermint, I had started working on extending the e2e framework for upgrade testing and perhaps we should endeavour to do the same here. We need to make sure this works alongside our versioning strategy. If we have long lived branches for our minor versions, we need to ensure that rolling upgrades happen smoothly for node operators. This is generally light-weight and thus could be tested on a per PR basis (in the same way you might detect protobuf breaking changes).

MSevey commented 1 year ago

I think an overarching goal around our testing should be to put together some written artefact across the teams which can really act as a handbook making it clear for any team member or for anyone interested what we test, how we test it, when we test things and so forth. There is no monolithic one solution fits all kind of approach to testing. It really requires a myriad of different solutions that juggle the tradeoffs between different forms of coverage, "introspectability", time and resource costs. Thinking about testing the same systematic way we think about protocols and writing "specifications" will produce more reliable software. Having a place where people can read about everything will make it easier for us to cover our blindspots.

One option to kick start this type of documentation could be a README in the test package that explains the different tooling and types of tests.

evan-forbes commented 8 months ago

we still have a dangling issues, but most have been ice-boxed