tarantool / test-run

Tarantool functional testing framework
14 stars 14 forks source link

Calculate parallel jobs based on available CPUs #435

Closed Totktonada closed 1 month ago

Totktonada commented 1 month ago

We're going to enable new ARM64 runners in CI, where test-run is invoked in a Docker container. At the same time, the runner is an LXD container created by the service provider. In this circumstances, the Docker container sees all the 128 online CPUs, but the runner may have only some of them available (depending on the pricing plan).

Python 3.3+ has a function to determine available CPUs, so we can reduce the parallelism to this value. We fall back to the online CPUs count on Python < 3.3.

After this change, test-run follows a CPU affinity mask set by taskset (and, I guess, by numactl).

The change is similar to replacing nproc --all to nproc.

The change only affects the default behavior, which can be overwritten by passing the --jobs (or -j) CLI option or using the TEST_RUN_JOBS environment variable.

Reported in https://github.com/tarantool/tarantool/pull/10102

coveralls commented 1 month ago

Coverage Status

coverage: 62.607% (+0.002%) from 62.605% when pulling f33c3243abddd0be7e37241d72b090a8206ff8d4 on Totktonada/calc-jobs-using-available-cpus into 84ebae5d041125ab3bb52514c1472db844c45f20 on master.

coveralls commented 1 month ago

Coverage Status

coverage: 62.607% (+0.002%) from 62.605% when pulling bec97bf2d593039134c41f6b4045ac39ac4393aa on Totktonada/calc-jobs-using-available-cpus into 84ebae5d041125ab3bb52514c1472db844c45f20 on master.

coveralls commented 1 month ago

Coverage Status

coverage: 62.607% (+0.002%) from 62.605% when pulling 4f926ba7fe1e4f173991393bf6bc7e88069d6f29 on Totktonada/calc-jobs-using-available-cpus into 84ebae5d041125ab3bb52514c1472db844c45f20 on master.

coveralls commented 1 month ago

Coverage Status

coverage: 62.675% (+0.07%) from 62.605% when pulling 66bef07baa78f918d629debbb244a8fb82734513 on Totktonada/calc-jobs-using-available-cpus into 84ebae5d041125ab3bb52514c1472db844c45f20 on master.