CTest -j parallel option creates mangled results with gtest_output directory usage

[apaprocki]

Current Behavior

The GTEST_OUTPUT environment variable (and command line argument) can be set to xml:directory/ to ensure each test is written to a separate file. This is required when using default CTest integration because CMake enumerates the tests in the resulting executable and then generates individual CTest rules for running each test individually. Without the directory output, the tests will all write to the same filename and nothing works as you'd expect. With the directory output, CTest's approach will only work if everything is run serially, due to this code:

// There could be a race condition if two or more processes are calling this
// function at the same time -- they could both pick the same filename.
FilePath FilePath::GenerateUniqueFileName(const FilePath& directory,
                                          const FilePath& base_name,
                                          const char* extension) {
  FilePath full_pathname;
  int number = 0;
  do {
    full_pathname.Set(MakeFileName(directory, base_name, number++, extension));
  } while (full_pathname.FileOrDirectoryExists());
  return full_pathname;
}

The comment above the function explains it all. If you run ctest -j 8 ... or similar, there are N processes racing for finding and creating the next numbered file, and the result is that you get two tests writing to the same XML file, producing overlapping output so that if a test with smaller output is written second, the XML data will having trailing characters from the prior test that wrote to the file. These XML files then fail validation when imported into systems that check the input. For example, Jenkins junit:

Failed to read test report file /path/to/gtest/output_36.xml
org.dom4j.DocumentException: Error on line 7 of document  : Content is not allowed in trailing section. Nested exception: Content is not allowed in trailing section.
    at org.dom4j.io.SAXReader.read(SAXReader.java:482)
    at org.dom4j.io.SAXReader.read(SAXReader.java:343)
        ...

Desired Behavior

I'm raising this issue in the hopes that the GTEST_OUTPUT/gtest_output format string can be modified to support variables provided by the GTest code. To me this can be solved in a generic manner for any test harness that works this way by either:

• a) Support writing a randomly generated number as a hexadecimal string suffix instead of the racy number++ (e.g., directory/footest_43a7f8d2.xml)
• b) Support writing the test name as the suffix instead of the racy number++ (e.g., directory/footest_BreathingTestForThing.xml

[gennadiycivil]

@apaprocki Thank you for this report. Sounds like this is about running googletest-based tests with CMake's CTest.

Anything related to CMake is community supported. We do not use it internally and we are not in position to check.

I would recommend that he best way to approach this would be to create a proper PR and submit it for consideration. Perhaps a test harness created to illustrate the desired behavior would make it easier for the community to contribute.

In any case, if there is a test harness it will take it easier for everyone to look.

HTH

[AlexanderStein]

I added a small testing repository in my space. This will cause sooner or later the problem.

I wonder what the correct fix is though. Unless you set a fixed file name always a new XML will be created.