search

teemtee / tmt

Test Management Tool
MIT License
84 stars 125 forks source link

Support parallel execution of tmt tests #2798

Open thrix opened 7 months ago

thrix commented 7 months ago

We now have a very beefy bare-metal (96 CPUs, 256Gi RAM) machine available for testing, and we would like to advise teams to use the beefy HW as much as possible. A common scenario for this machines will be teams running local VMs for their testing. Once they onboard to tmt including test metadata, they would want to run the tests in parallel.

For these cases, it seems it would be valuable if tmt would provide a way to parallelize test execution. Note that currently one possible issue here is that each test run execution is run in the same test dir, teams have to implement workarounds like https://github.com/RHSecurityCompliance/contest/pull/121/commits/fd519da1c2e2140d386a6df83d705df8f961ed85 to workraound the problem.

comps commented 7 months ago

Note that the linked https://github.com/RHSecurityCompliance/contest/commit/fd519da1c2e2140d386a6df83d705df8f961ed85 is functionally equivalent to

    rlPhaseStartSetup
        rlRun "TmpDir=\$(mktemp -d)" 0 "Creating tmp directory"
        rlRun "pushd $TmpDir"
    rlPhaseEnd

    ...

    rlPhaseStartCleanup
        rlRun "popd"
        rlRun "rm -r $TmpDir" 0 "Removing tmp directory"
    rlPhaseEnd

which most Beakerlib users are likely familiar with.

LecrisUT commented 7 months ago

Related to #2755. Over there I point out that it can be disastrous in MPI tests if the inner test parallelization is not considered. I like ctest approach where you can define individual test's processor and other resources and then it queues accordingly, where the -j accounts both for test mpi parallelization and ctest workers combined. Tmt should have a similar interface.

Concrete numbers: 16h vs 30min