Some tests are written in a way that there are cross-dependencies such that failure in one test might lead to a failure in a dependent test further down the pipeline.
Also, some tests replicate testing edge cases on the parachain/clients that are unrelated to the lib code. These tests do not provide extra assurance about the code but rather duplicate testing efforts that are on the parachain/clients (or should be moved there if they don't already exist).
This leads to two main issues:
For regular PRs, there's a significant wait time for the pipeline. This has negative impact on the developer experience and limits the number of times we can run the pipeline in a given day - thus reducing code velocity.
For emergency fixes, the pipeline often needs to be skipped as critical bug testing must happen outside the 45 min plus pipeline for fix deployment. This circumvents the point of having integration tests.
Describe the solution you'd like
The overall goal is to reduce the lib pipeline to under 10 minutes. Ideally the pipeline should be even faster than that but that seems like an acceptable first goal.
To achieve this, we need to tackle a couple of things. This is an open-ended list and can be extended.
[ ] Initialize the parachain with the state at the end of the setup tests: https://github.com/interlay/interbtc-api/blob/master/test/integration/parachain/staging/setup/initialize.test.ts. If the lib does not have to go about time consuming setup but rather the initial chain state is already correctly set, we can save quite some time there. Moreover, the functions tested there are not functions that we would expect to be called form the lib (registering foreign assets, setting vault rewards, ...)
@gregdhill @peterslany @bvotteler should we close this issue? Not sure we did all of the above, but pipeline is now below 10 minutes so I'd be inclined to close this issue.
Is your feature request related to a problem? Please describe. The lib integration tests run for 45 min on the kintsugi version (https://github.com/interlay/interbtc-api/actions/workflows/integration-tests-kint.yml) and for more than 90 minutes on the interlay version (https://github.com/interlay/interbtc-api/actions/workflows/integration-tests-intr.yml).
Some tests are written in a way that there are cross-dependencies such that failure in one test might lead to a failure in a dependent test further down the pipeline.
Also, some tests replicate testing edge cases on the parachain/clients that are unrelated to the lib code. These tests do not provide extra assurance about the code but rather duplicate testing efforts that are on the parachain/clients (or should be moved there if they don't already exist).
This leads to two main issues:
Describe the solution you'd like
The overall goal is to reduce the lib pipeline to under 10 minutes. Ideally the pipeline should be even faster than that but that seems like an acceptable first goal.
To achieve this, we need to tackle a couple of things. This is an open-ended list and can be extended.