Improve overall test time

[eagr]

The rough idea of what we talked about. :)) @liamaharon

[liamaharon]

Looking good, thanks!

It would be nice if we could specify a block number we'd like it to run to before proceeding as well.

[eagr]

Could you elaborate a bit on "a block number we'd like it to run to"?

[liamaharon]

Poll the current block of the spun up chain and yield after a block has been mined.

[eagr]

You mean something like this?

async fn block_hash(from: u64, to: u64, url: &str, timeout: u64) -> Result<Vec<Hash>, String>;

[eagr]

By grouping the integration tests, on my machine, it could save ~70s and ~1GB disk space. In case it's not obvious from the report, this essential reduces the amount of build targets, therefore reduces the overall amount of work (linking) and generated artifacts.

Original

Grouped

[eagr]

Thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: 
GlobalLoggerError(SetLoggerError(SetLoggerError(())))',
/Users/eagr/.cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/1d5d4a4/substrate/test-utils/cli/src/lib.rs:66

I suspect this is due to we're running the tests in parallel now, spinning up multiple dev nodes at the same time. Any ideas?

[eagr]

This would solve the issue, and the tests also run faster this way. But I'm not sure whether this is what you want. The logs are all mixed up as the tests run in parallel now.

[eagr]

Currently, the tests run sequentially in separate processes. There are no problems because by the time the next test runs, the dev node has exited with the previous test process.

After grouping the tests run in one process concurrently (different threads). The problem originates from setting the logger multiple times in a process, which is disallowed by the tracing crate.

https://docs.rs/tracing-log/latest/tracing_log/log_tracer/struct.SetLoggerError.html

Making the tests run sequentially (either with lock or --test-threads=1) wouldn't work, unless there is a way to force the dev node to exit at the end of a test.

TL;DR Grouping the tests has the benefits of faster compilation (-1min) and execution (-3min), but if running tests in parallel on the same node isn't what you want, let's revert the changes.