search

rust-lang / crater

Run experiments across parts of the Rust ecosystem!
https://crater.rust-lang.org
642 stars 90 forks source link

Limit available parallelism to 1 per 2GB of RAM #725

Open riking opened 8 months ago

riking commented 8 months ago

This is a proposed solution to the OOM class of issues mentioned in #589 , in particular the pattern where a large number of integration tests are launched in parallel.

Common CPU-to-RAM ratios for cloud provider VMs are 1:2GB, 1:4GB, and 1:8GB. Because there is currently no parallelism limit imposed, use the most generous ratio for the initial implementation.

Proposed Implementation:

src/runner/test.rs is updated to calculate the total memory available to the nearest limiting control group or whole machine, divide this number by 2 gigabytes and provide the result to parallelism-limiting mechanisms.

Each relevant cargo command accepts -jNUM, but cargo test could also specify --test-threads=NUM for the default test runner to limit its parallelism as well.

riking commented 8 months ago

actually, this should only apply to the test runner phase, right? We still want maximum parallel during the build of the tests.