google / gtest-parallel

Run Google Test suites in parallel.
Apache License 2.0
421 stars 104 forks source link

add support for {#} similar to parallel #89

Open 1480c1 opened 2 years ago

1480c1 commented 2 years ago

allows for --gtest_output=xml:reports/report_{#}.xml to be used instead of relying on googletest itself to lock a unique filename and using that

usage would be something like

python \
    gtest_parallel.py \
    --print_test_times \
    ./unit_tests \
    -- \
    --gtest_output="xml:reports/{#}.xml"

and then uploading the reports/*.xml files for junit reports or using something like junitparser to merge them together.

Formatted using yapf, so that's where the extra spaces come from.

1480c1 commented 2 years ago

This mainly targets a different direction than https://github.com/google/gtest-parallel/pull/81 as rather than attempting to merge the resulting output inside the script, it just allows the script to have the xmls be produced to different files, but does allow the script to be used inside gitlab-ci by using something like

  artifacts:
    when: always
    reports:
      junit: reports/*.xml

in their yml file

pbos commented 2 years ago

Intuitively I think #81 would make more sense as an approach to support, but it stalled out. Would this solve something else than #81 does? At first glance it looks like a workaround for #81 having stalled.

This part to me was somewhat worrying and I haven't seen anything since that comment:

"The merging of the xml fails, if one of the tests crashes the test itself. Which happens for me with glog and CHECK(...). This creates a SIGABRT. The merge fails because no xml is created for that test."

1480c1 commented 2 years ago

Would this solve something else than https://github.com/google/gtest-parallel/pull/81 does? At first glance it looks like a workaround for https://github.com/google/gtest-parallel/pull/81 having stalled.

Well, the original intention for this PR was something of a workaround for #81 being stalled, but in a sense, is more general since it does not care about if the {#} is in --gtest_output= or not and can be used elsewhere in the command line and can be used in --gtest_filter= if for some reason you have a group of tests that uses numbers in its name.

This was originally inspired by parallel since at one point I fiddled around with filtering the output of --gtest_list_tests with awk to get the full test names and then piping that into parallel to then call the unit test binaries directly with --gtest_output="xml:reports/{#}.xml" to replicate gtest_parallel.py, but I found out that due to the sheer number of small tests we have, it often took longer for the init than it actually took for the test.

The primary reason why I did not look into continuing #81 is simply because of my lack of python skill, so I have no confidence in being able to complete what that PR attempts to accomplish in the same time period as what it took for this modification

pbos commented 2 years ago

I think the real solution should end up solving #81. I think this is better suited on a local branch, I'm wary of adding parameter support in general for things that are not supported in upstream gtest (and am very aware that I've done so in the past, imo -r shouldn't exist, but rather be just --gtest_repeat).