CI improvement recommendations meta-issue

[appetrosyan]

About

This is a blanket issue for collecting proposals on how to make the CI for iroha2-dev faster and more reliable.

@i1i1 @GaroRobe @s8sato @KalitaAlexey your input is welcome.

[i1i1]

Make docker image for check workflow in ci

It would remove installing cargo lints and nightly toolchain for formatting.

[i1i1]

Cache ci runs

If files didnt change, there is no need to rerun ci job.

Issues

• Managing and maintaining dependent files hashes for each test workflow.

[i1i1]

Removing docker tests with iroha_client_cli

Docker tests are the slowest part of CI and they dont actually test anything apart from:

• docker image build
• iroha peer startup flags
• docker compose setup

I guess we can move it to iroha2 branch and just fix it there. Also we need to update it for iroha_python library. I guess we can even build iroha_python library there

[appetrosyan]

Separating lightweight tests

When any of the heavier tests fails, one has to re-run the entire testing workflow including static checks.

So we would have

• DCO
• Code formatting and static checks
• Lightweight unit testing of the crates
• integration testing
• Docker tests
• Telemetry tests

This will allow us to re-run just the workflows that failed. As an option it might be a good idea to stop the subsequent workflows if the earlier e.g. format checks have failed.

[i1i1]

Cache rust builds even at workflow failure

[i1i1]

Prebuild everything before check workflows run

Make all check workflows depend on smth like:

cargo build --all --tests --benches

[appetrosyan]

Migrate docker tests to Iroha Python.

iroha-CLI can be replaced with a lightweight script that does the same, and is also officially supported. This doubles as an integration test and highlights API-breaking changes.

[appetrosyan]

Replace iroha_CLI with a separate program using a C API

this will allow lightweight testing of iroha facilities using a program that doesn’t strictly need to be rebuilt every time. This will discourage external-API changes, and encourage more extensive and reliable testing.

[i1i1]

Let CI format and fix the lints on a merge

Make workflow which does the following:

cargo +nightly fmt
cargo lints clippy  --all --tests --fix --benches
git commit -a --amend --no-edit
git push -f

[appetrosyan]

Let CI format and fix the lints on a merge

Make workflow which does the following:

cargo +nightly fmt
cargo lints clippy  --all --tests --fix --benches
git commit -a --amend --no-edit
git push -f

totally onboard for applying formatting changes.

I'm a little iffy of automatically applying clippy suggestions. They often require manual intervention.

[appetrosyan]

@eadventurous we would greatly appreciate your input on these issues.

[i1i1]

I'm a little iffy of automatically applying clippy suggestions. They often require manual intervention.

Most of them (like remove clone) don't though :)

It wont hurt anyway to add clippy --fix

[i1i1]

Move unstable network tests into separate workflow

[appetrosyan]

Create a separate staging branch for bulk merge.

Currently even trivial changes require running a full suite of tests. iroha2-dev could retain the current set of tests, but instead of merging our work directly into iroha2-dev we could merge into iroha2-dev-staging which has faster and more minimal checks. We open a PR once a week (or once a month) and enable all other tests, including the more expensive ones.

This would also allow us to produce benchmarking results with some regularity.

[appetrosyan]

Consider using the mold linker

mold can significantly speed up the linking process of the build (which in our case of having multiple custom crates is likely a bottleneck).

[appetrosyan]

Consider using smaller Docker images

Most of our unrealiable tests run out of memory. Due to the way docker containers work, it might be improved by using a more minimal customised docker image that contains our dependencies and does not contain any extraneous packages/systems.

[appetrosyan]

Consider granular dependency based-testing.

• If only md files changed, the only things that need to run are add_label and DCO.

• If only the json files changed, we don't need to re-run the source based unit tests (which we don't differentiate from other tests, even though I think that we should).

• If a crate changed, only its, its dependencies' and the integration tests need to be re-run. This saves on running many unit tests for crates like client_cli.

[appetrosyan]

Consider parallelising the long running workflows.

The telemetry tests consist of several paralleliseable chunks. For example make a debug build and print its telemetry as a separate job from making a release build and printing telemetry there.

We gain speed, because there are fewer long-running checks than we have runners (2 vs 4).

[e-ivkov]

Future telemetry also doesn't seem to me like a necessary workflow. We might use this feature when debugging but I don't think it should be on each PR.

Maybe we should just move it to pipeline that is run after merge.

[appetrosyan]

Use the cranelift compiler https://github.com/bjorn3/rustc_codegen_cranelift/blob/master/docs/usage.md

[appetrosyan]

Use smarter cache. https://github.com/Swatinem/rust-cache