Add parallel tests execution to CI

smiasojed commented 7 months ago

Summary

Closes #_

[ ] y/n | Does it introduce breaking changes?
[ ] y/n | Is it dependant on the specific version of cargo-contract or pallet-contracts?

Description

Add nextest to the workflow where parallel test execution makes sense. In cases where parallel execution of tests cannot be used, split the tests between different machines.

Checklist before requesting a review

[ ] My code follows the style guidelines of this project
[ ] I have added an entry to CHANGELOG.md
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] Any dependent changes have been merged and published in downstream modules

smiasojed commented 7 months ago

Time for test job decreased by 50%, from 18min to 9 min

There is still place for improvements:

        PASS [  25.236s] ink::compile_tests ui_tests_selector_bytes_fail
        PASS [  25.746s] ink::compile_tests ui_tests_blake2b_fail
        PASS [  26.222s] ink::compile_tests ui_tests_scale_derive_fail
        PASS [  26.738s] ink::compile_tests ui_tests_selector_id_fail
        PASS [  27.784s] ink::compile_tests ui_tests_pay_with_call_pass
        PASS [  28.315s] ink::compile_tests ui_tests_event_fail
        PASS [  30.498s] ink::compile_tests ui_tests_event_pass
        PASS [  31.292s] ink::compile_tests ui_tests_contract_fail
        PASS [  32.972s] ink::compile_tests ui_tests_scale_derive_pass
        PASS [  34.528s] ink::compile_tests ui_tests_blake2b_pass
        PASS [  35.363s] ink::compile_tests ui_tests_chain_extension_pass
        PASS [  36.986s] ink::compile_tests ui_tests_selector_bytes_pass
        SLOW [> 60.000s] ink::compile_tests ui_tests_contract_pass
        SLOW [> 60.000s] ink::compile_tests ui_tests_storage_item_pass
        PASS [  79.644s] ink::compile_tests ui_tests_contract_pass
        PASS [  85.505s] ink::compile_tests ui_tests_storage_item_pass

Still, each test like ink::compile_tests ui_tests_contract_pass consists of tens of subtests that cannot be run in parallel. These tests needs to be refactored - to be addressed in separate task

smiasojed commented 7 months ago

Each end-to-end test takes between 50 to 90 seconds. Currently, they are executed iteratively because end-to-end tests build contracts during test execution, and the same contracts are rebuilt by multiple tests instead of reusing them, causing a race condition.

github-actions[bot] commented 7 months ago

🦑 📈 ink! Example Contracts ‒ Changes Report 📉 🦑

These are the results when building the integration-tests/* contracts from this branch with cargo-contract and comparing them to ink! master:

Contract	Upstream Size (kB)	PR Size (kB)	Change
call-builder-return-value	9.249	9.249	:heavy_minus_sign:
call-runtime	2.071	2.071	:heavy_minus_sign:
combined-extension	2.132	2.132	:heavy_minus_sign:
conditional-compilation	1.502	1.502	:heavy_minus_sign:
contract-storage	7.58	7.58	:heavy_minus_sign:
contract-terminate	1.369	1.369	:heavy_minus_sign:
contract-transfer	1.731	1.731	:heavy_minus_sign:
cross-contract-calls	7.732	7.732	:heavy_minus_sign:
cross-contract-calls/other-contract	1.595	1.595	:heavy_minus_sign:
custom-allocator	7.787	7.787	:heavy_minus_sign:
custom-environment	2.158	2.158	:heavy_minus_sign:
dns	7.355	7.355	:heavy_minus_sign:
e2e-call-runtime	1.32	1.32	:heavy_minus_sign:
e2e-runtime-only-backend	1.901	1.901	:heavy_minus_sign:
erc1155	14.345	14.345	:heavy_minus_sign:
erc20	6.955	6.955	:heavy_minus_sign:
erc721	10.044	10.044	:heavy_minus_sign:
events	5.27	5.27	:heavy_minus_sign:
flipper	1.651	1.651	:heavy_minus_sign:
incrementer	1.516	1.516	:heavy_minus_sign:
lang-err-integration-tests/call-builder-delegate	2.65	2.65	:heavy_minus_sign:
lang-err-integration-tests/call-builder	5.571	5.571	:heavy_minus_sign:
lang-err-integration-tests/constructors-return-value	1.997	1.997	:heavy_minus_sign:
lang-err-integration-tests/contract-ref	5.062	5.062	:heavy_minus_sign:
lang-err-integration-tests/integration-flipper	1.827	1.827	:heavy_minus_sign:
lazyvec-integration-test	4.66	4.66	:heavy_minus_sign:
mapping-integration-tests	8.036	8.036	:heavy_minus_sign:
mother	12.753	12.753	:heavy_minus_sign:
multi-contract-caller	6.654	6.654	:heavy_minus_sign:
multi-contract-caller/accumulator	1.388	1.388	:heavy_minus_sign:
multi-contract-caller/adder	1.922	1.922	:heavy_minus_sign:
multi-contract-caller/subber	1.942	1.942	:heavy_minus_sign:
multisig	21.871	21.871	:heavy_minus_sign:
payment-channel	5.742	5.742	:heavy_minus_sign:
psp22-extension	7.083	7.083	:heavy_minus_sign:
rand-extension	2.977	2.977	:heavy_minus_sign:
sr25519-verification	1.154	1.154	:heavy_minus_sign:
static-buffer	2.578	2.578	:heavy_minus_sign:
trait-dyn-cross-contract-calls	2.899	2.899	:heavy_minus_sign:
trait-dyn-cross-contract-calls/contracts/incrementer	1.557	1.557	:heavy_minus_sign:
trait-erc20	7.331	7.331	:heavy_minus_sign:
trait-flipper	1.502	1.502	:heavy_minus_sign:
trait-incrementer	1.626	1.626	:heavy_minus_sign:
upgradeable-contracts/delegator	3.96	3.96	:heavy_minus_sign:
upgradeable-contracts/delegator/delegatee	1.641	1.641	:heavy_minus_sign:
upgradeable-contracts/delegator/delegatee2	1.641	1.641	:heavy_minus_sign:
upgradeable-contracts/set-code-hash-migration	1.755	1.755	:heavy_minus_sign:
upgradeable-contracts/set-code-hash-migration/migration	1.462	1.462	:heavy_minus_sign:
upgradeable-contracts/set-code-hash-migration/updated-incrementer	1.909	1.909	:heavy_minus_sign:
upgradeable-contracts/set-code-hash	1.755	1.755	:heavy_minus_sign:
upgradeable-contracts/set-code-hash/updated-incrementer	1.733	1.733	:heavy_minus_sign:
wildcard-selector	2.858	2.858	:heavy_minus_sign:

Link to the run | Last update: Thu Mar 21 11:03:46 CET 2024

smiasojed commented 7 months ago

The examples-test job is split into 4 parallel jobs: linear duration decreased from 36 to 12 minutes, but when we sum up the 4 parallel jobs, we get 48 minutes (the cost of launching and compilation of common crates on each machine)

The option here is to use nextest, where the compilation is run on one machine and execute tests in parallel on others. However, there are two issues: e2e tests cannot be executed in parallel due to a contract build issue, and even when precompiled data is shared, compilation still occurs on machines meant for conducting tests. To make it work, we also need to have integration tests in one workspace. This should be considered in the future when the compilation issue is fixed. For now, there is no gain because we cannot use parallel execution on machines with this issue.

codecov-commenter commented 7 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 61.25%. Comparing base (232f7fd) to head (5f09dd5). Report is 2 commits behind head on master.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #2155 +/- ## ========================================== + Coverage 54.20% 61.25% +7.04% ========================================== Files 231 139 -92 Lines 7204 5709 -1495 Branches 3153 2421 -732 ========================================== - Hits 3905 3497 -408 + Misses 3299 2212 -1087 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

smiasojed commented 7 months ago

Total time has changed from 70 to 50 minutes Billable time decreased by 10 minutes

Now, on the critical path, we have the ink build and contracts measurements, both of which take 29 minutes - to be optimized as separate PR

use-ink / ink