ros2 / ci

ROS 2 CI Infrastructure
http://ci.ros2.org/
Apache License 2.0
48 stars 30 forks source link

:farmer: Nightly coverage job failing in subprocess run #770

Open Crola1702 opened 5 months ago

Crola1702 commented 5 months ago

Rolling nightly coverage job is failing to generate coverage reports, killing node ssh-agent and generating a lot of java exceptions due to the closed connection.

This problem is happening since the change to use Noble by default on this job.

Reference builds:

The failure is thrown in: https://github.com/ros2/ci/blob/b6cd705b12547a18db6104c20077cc0dba66dd1b/ros2_batch_job/__main__.py#L295-L298 and it's the following:

# BEGIN SUBSECTION: coverage analysis
['lcov', '--capture', '--directory', 'build', '--output', 'build/coverage.info']
Capturing coverage data from build
geninfo cmd: '/usr/bin/geninfo build --output-filename build/coverage.info --memory 0'
Found gcov version: 13.2.0
Using intermediate gcov format
Writing temporary data to /tmp/geninfo_datTvIy
Scanning build for .gcda files ...
Found 3093 data files in build
Processing build/test_tf2/CMakeFiles/test_message_filter.dir/test/test_message_filter.cpp.gcda
geninfo: WARNING: using JSON module "JSON::PP" - which is much slower than some alternatives.  Consider installing one of JSON::XS or Cpanel::JSON::XS
geninfo: ERROR: mismatched end line for _ZN31MessageFilter_noTransforms_Test8TestBodyEv at /home/jenkins-agent/workspace/nightly_linux_coverage/ws/src/ros2/geometry2/test_tf2/test/test_message_filter.cpp:79: 79 -> 100   
    (use "geninfo --ignore-errors mismatch ..." to bypass this error)
Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux_coverage/run_ros2_batch.py", line 32, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/jenkins-agent/workspace/nightly_linux_coverage/ros2_batch_job/__main__.py", line 154, in main                                                                            
    return run(args, build_function, blacklisted_package_names=blacklisted_package_names)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jenkins-agent/workspace/nightly_linux_coverage/ros2_batch_job/__main__.py", line 689, in run
    rc = build_function(args, job, colcon_script)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jenkins-agent/workspace/nightly_linux_coverage/ros2_batch_job/__main__.py", line 386, in build_and_test
    process_coverage(args, job)
  File "/home/jenkins-agent/workspace/nightly_linux_coverage/ros2_batch_job/__main__.py", line 280, in process_coverage
    subprocess.run(cmd, check=True)
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['lcov', '--capture', '--directory', 'build', '--output', 'build/coverage.info']' returned non-zero exit status 1.
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 1211 killed;

Mismatched line error points to: https://github.com/ros2/geometry2/blob/c42ad86519853d81ad630e48111ae82afcbf61ec/test_tf2/test/test_message_filter.cpp#L79-L100