Some ASAN failures not reflected in junit results

I've noticed that some ASAN test failures are not reflected in the JUNIT result files. For example, the following build is marked as unstable with two test failures:

https://build.osrfoundation.org/view/asan/job/ignition_common-ci_asan-ign-common3-focal-amd64/10/testReport/

From the console log, however, it is evident that more than two tests failed:

The following tests FAILED:
    [0;31m 41 - INTEGRATION_encoder_timing (Failed)[0;0m
    [0;31m 45 - INTEGRATION_video_encoder (Failed)[0;0m
    [0;31m 56 - UNIT_AudioDecoder_TEST (Failed)[0;0m
    [0;31m 58 - UNIT_VideoEncoder_TEST (Failed)[0;0m
    [0;31m 68 - UNIT_ColladaExporter_TEST (Failed)[0;0m
    [0;31m 70 - UNIT_ColladaLoader_TEST (Failed)[0;0m
    [0;31m 78 - UNIT_Image_TEST (Failed)[0;0m
    [0;31m 84 - UNIT_Mesh_TEST (Failed)[0;0m
    [0;31m 86 - UNIT_OBJLoader_TEST (Failed)[0;0m
    [0;31m 92 - UNIT_SubMesh_TEST (Failed)[0;0m
Errors while running CTest

The UNIT_SubMesh_TEST console output is illustrative as the test appears to pass and presumably writes a successful junit file, but then writes out ASAN errors at the very end and returns an error code.

test 92
      Start 92: UNIT_SubMesh_TEST

92: Test command: /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/build/bin/UNIT_SubMesh_TEST "--gtest_output=xml:/home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/build/test_results/UNIT_SubMesh_TEST.xml"
92: Test timeout computed to be: 240
92: [==========] Running 2 tests from 1 test suite.
92: [----------] Global test environment set-up.
92: [----------] 2 tests from SubMeshTest
92: [ RUN      ] SubMeshTest.SubMesh
92: [1;31m[Err] [SubMesh.cc:400] [0m[1;31mIndex too large[0m[1;31m[0m
92: [1;31m[Err] [SubMesh.cc:218] [0m[1;31mIndex too large[0m[1;31m[0m
92: [1;31m[Err] [SubMesh.cc:199] [0m[1;31mIndex too large[0m[1;31m[0m
92: [1;31m[Err] [SubMesh.cc:283] [0m[1;31mIndex too large[0m[1;31m[0m
92: [1;31m[Err] [SubMesh.cc:230] [0m[1;31mIndex too large[0m[1;31m[0m
92: [1;31m[Err] [SubMesh.cc:363] [0m[1;31mIndex too large[0m[1;31m[0m
92: [1;31m[Err] [SubMesh.cc:324] [0m[1;31mIndex too large[0m[1;31m[0m
92: [1;31m[Err] [SubMesh.cc:387] [0m[1;31mIndex too large[0m[1;31m[0m
92: [1;31m[Err] [SubMesh.cc:375] [0m[1;31mIndex too large[0m[1;31m[0m
92: [1;31m[Err] [SubMesh.cc:559] [0m[1;31mNo vertices or indices[0m
92: [       OK ] SubMeshTest.SubMesh (3 ms)
92: [ RUN      ] SubMeshTest.Volume
92: [1;31m[Err] [SubMesh.cc:732] [0m[1;31mVolume calculation can only be accomplished on a triangulated [0m[1;31m mesh.[0m
92: [1;31m[Err] [SubMesh.cc:727] [0m[1;31mThe number of indices is not a multiple of three.[0m
92: [       OK ] SubMeshTest.Volume (390 ms)
92: [----------] 2 tests from SubMeshTest (393 ms total)
92: 
92: [----------] Global test environment tear-down
92: [==========] 2 tests from 1 test suite ran. (393 ms total)
92: [  PASSED  ] 2 tests.
92: 
92: =================================================================
92: ==3728==ERROR: LeakSanitizer: detected memory leaks
92: 
92: Direct leak of 72 byte(s) in 1 object(s) allocated from:
92:     #0 0x7fcd8f6ea787 in operator new[](unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:107
92:     #1 0x7fcd8f4f90ba in ignition::common::SubMesh::FillArrays(double**, int**) const /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/graphics/src/SubMesh.cc:569
92:     #2 0x55e14f2fa403 in SubMeshTest_SubMesh_Test::TestBody() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/graphics/src/SubMesh_TEST.cc:280
92:     #3 0x55e14f43f107 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2437
92:     #4 0x55e14f43f107 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2473
92:     #5 0x55e14f40deca in testing::Test::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2512
92:     #6 0x55e14f40e575 in testing::Test::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2503
92:     #7 0x55e14f40e575 in testing::TestInfo::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2688
92:     #8 0x55e14f40eb22 in testing::TestInfo::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2662
92:     #9 0x55e14f40eb22 in testing::TestSuite::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2820
92:     #10 0x55e14f410596 in testing::TestSuite::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2800
92:     #11 0x55e14f410596 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:5342
92:     #12 0x55e14f440247 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2437
92:     #13 0x55e14f440247 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2473
92:     #14 0x55e14f410f0d in testing::UnitTest::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:4929
92:     #15 0x55e14f2e446f in RUN_ALL_TESTS() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/include/gtest/gtest.h:2473
92:     #16 0x55e14f2e446f in main /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/graphics/src/SubMesh_TEST.cc:509
92:     #17 0x7fcd8eb3a082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082)
92: 
92: Direct leak of 12 byte(s) in 1 object(s) allocated from:
92:     #0 0x7fcd8f6ea787 in operator new[](unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:107
92:     #1 0x7fcd8f4f914a in ignition::common::SubMesh::FillArrays(double**, int**) const /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/graphics/src/SubMesh.cc:570
92:     #2 0x55e14f2fa403 in SubMeshTest_SubMesh_Test::TestBody() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/graphics/src/SubMesh_TEST.cc:280
92:     #3 0x55e14f43f107 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2437
92:     #4 0x55e14f43f107 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2473
92:     #5 0x55e14f40deca in testing::Test::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2512
92:     #6 0x55e14f40e575 in testing::Test::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2503
92:     #7 0x55e14f40e575 in testing::TestInfo::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2688
92:     #8 0x55e14f40eb22 in testing::TestInfo::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2662
92:     #9 0x55e14f40eb22 in testing::TestSuite::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2820
92:     #10 0x55e14f410596 in testing::TestSuite::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2800
92:     #11 0x55e14f410596 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:5342
92:     #12 0x55e14f440247 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2437
92:     #13 0x55e14f440247 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:2473
92:     #14 0x55e14f410f0d in testing::UnitTest::Run() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/src/gtest.cc:4929
92:     #15 0x55e14f2e446f in RUN_ALL_TESTS() /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/test/gtest/include/gtest/gtest.h:2473
92:     #16 0x55e14f2e446f in main /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/ign-common/graphics/src/SubMesh_TEST.cc:509
92:     #17 0x7fcd8eb3a082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082)
92: 
92: SUMMARY: AddressSanitizer: 84 byte(s) leaked in 2 allocation(s).
92/97 Test #92: UNIT_SubMesh_TEST ...........................***Failed    0.86 sec
test 93
      Start 93: check_UNIT_SubMesh_TEST

93: Test command: /usr/bin/python3.8 "/usr/share/ignition/ignition-cmake2/tools/check_test_ran.py" "/home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/build/test_results/UNIT_SubMesh_TEST.xml"
93: Test timeout computed to be: 1500
93: Checking for test results in /home/jenkins/workspace/ignition_common-ci_asan-ign-common3-focal-amd64/build/test_results/UNIT_SubMesh_TEST.xml
93/97 Test #93: check_UNIT_SubMesh_TEST .....................   Passed    0.02 sec

We should have smarter logic to detect these failures.

gazebo-tooling / release-tools

Some ASAN failures not reflected in junit results #820