Buildfarmer log 2024/05/20 - 2024/05/27

osrf / buildfarm-tools

2 stars 2 forks source link

Buildfarmer log 2024/05/20 - 2024/05/27 #53

Closed github-actions[bot] closed 3 months ago

github-actions[bot] commented 3 months ago

Previous log #51

github-actions[bot] commented 3 months ago

May 20

Build regressions

``` ```

Test regressions

``` Excluding 'performance, connext, fastrtps-dynamic' jobs Issue #1 in nightly_linux-aarch64_repeated - Failing 84.62% in 11 out of 13 builds rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice projectroot.test_sequential_compression_writer (also in: "nightly_win_rep" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #2 in nightly_win_rep - Failing 54.55% in 6 out of 11 builds projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) Issue #3 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) Issue #7 in nightly_win_rep - Failing 45.45% in 5 out of 11 builds rclcpp.test_executor.gtest.missing_result (also in: "nightly_win_rel" job) Issue #8 in gz_rendering-main-win - Failing 100.0% in 4 out of 4 builds MeshTest.NormalMapWithoutTexCoord (also in: "gz_rendering-8-win" job) (also in: "gz_rendering-7-win" job) Issue #9 in nightly_linux-rhel_repeated - Failing 30.77% in 4 out of 13 builds projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp Issue #10 in nightly_linux_repeated - Failing 33.33% in 4 out of 12 builds ros2param.ros2param.test.test_verb_load.test_verb_load (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED) (also in: "nightly_linux-rhel_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED) (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED) Issue #11 in nightly_win_rep - Failing 36.36% in 4 out of 11 builds rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock projectroot.test.rclcpp.test_clock (also in: "nightly_win_rel" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) Issue #12 in nightly_linux_repeated - Failing 25.0% in 3 out of 12 builds test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp Issue #13 in nightly_win_rel - Failing 27.27% in 3 out of 11 builds sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes (also in: "nightly_win_rep" job) (also in: "nightly_win_deb" job) Issue #14 in nightly_win_rep - Failing 27.27% in 3 out of 11 builds test_rclcpp.test_timeout_subscriber.timeout_subscriber Issue #15 in nightly_linux-rhel_repeated - Failing 23.08% in 3 out of 13 builds demo_nodes_cpp.TestExecutablesTutorial.test_processes_output (also in: "nightly_win_rep" job) (also in: "nightly_win_deb" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux_debug" job) (also in: "nightly_linux-rhel_repeated" job) Issue #16 in nightly_linux-rhel_repeated - Failing 15.38% in 2 out of 13 builds test_rclcpp.TestNNodes.test_10_nodes (also in: "nightly_win_rep" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #17 in nightly_linux_repeated - Failing 16.67% in 2 out of 12 builds projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp Issue #18 in nightly_win_rep - Failing 18.18% in 2 out of 11 builds rclcpp_action.TestServer.publish_status_succeeded rclcpp_action.TestServer.publish_status_canceling rclcpp_action.TestServer.publish_status_canceled rclcpp_action.TestServer.publish_status_accepted rclcpp_action.TestServer.publish_status_aborted rclcpp_action.TestServer.publish_feedback rclcpp_action.TestServer.handle_goal_called rclcpp_action.TestServer.handle_cancel_unknown_goal rclcpp_action.TestServer.handle_cancel_terminated_goal rclcpp_action.TestServer.handle_cancel_reject rclcpp_action.TestServer.handle_cancel_called rclcpp_action.TestServer.handle_accepted_called rclcpp_action.TestServer.get_result_deferred rclcpp_action.TestServer.get_result rclcpp_action.TestServer.deferred_execution rclcpp_action.TestServer.construction_and_destruction_wait_set_error rclcpp_action.TestServer.construction_and_destruction_sub_node rclcpp_action.TestServer.construction_and_destruction_server_init_error rclcpp_action.TestServer.construction_and_destruction_callback_group projectroot.test_server Issue #19 in nightly_linux-rhel_repeated - Failing 15.38% in 2 out of 13 builds test_rclcpp.TestNNodesAfterShutdown.test_10_nodes projectroot.test_n_nodes__rmw_fastrtps_cpp (also in: "nightly_win_rep" job) (also in: "nightly_linux-rhel_repeated" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #20 in nightly_win_rep - Failing 18.18% in 2 out of 11 builds projectroot.test_client (also in: "nightly_win_deb" job) (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job) (also in: "Jci__nightly-connext_ubuntu_noble_amd64" job) Issue #21 in gz_sim-ign-gazebo6-win - Failing 33.33% in 1 out of 3 builds INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) Issue #22 in nightly_linux-rhel_repeated - Failing 7.69% in 1 out of 13 builds sros2.test.sros2.commands.security.verbs.test_generate_policy.test_generate_policy (also in: "nightly_win_rep" job) (also in: "nightly_win_rel" job) (also in: "nightly_linux_repeated" job) Issue #23 in nightly_linux-rhel_repeated - Failing 7.69% in 1 out of 13 builds ros2param.ros2param.test.test_verb_dump.test_verb_dump (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED) (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/rmw_fastrtps/issues/531 - CLOSED | https://github.com/ros2/ros2cli/issues/861 - CLOSED) Issue #24 in nightly_linux_iron_coverage - Failing 8.33% in 1 out of 12 builds launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) (also in: "Ici__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) (also in: "Hci__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) Issue #25 in nightly_linux-rhel_repeated - Failing 7.69% in 1 out of 13 builds test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #26 in nightly_linux-rhel_repeated - Failing 7.69% in 1 out of 13 builds projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #27 in nightly_win_rep - Failing 9.09% in 1 out of 11 builds rclcpp_action.TestServer.construction_and_destruction_after_node rclcpp_action.TestServer.construction_and_destruction rclcpp_action.TestClientAgainstServer.send_rcl_errors rclcpp_action.TestClientAgainstServer.execute_rcl_errors rclcpp_action.TestClientAgainstServer.deadlock_in_callbacks rclcpp_action.TestClientAgainstServer.async_cancel_some_goals_with_callback rclcpp_action.TestClientAgainstServer.async_cancel_some_goals projectroot.test_showimage_cam2image__rmw_fastrtps_cpp projectroot.gtest_timeout_subscriber__rmw_cyclonedds_cpp image_tools.TestExecutablesDemo.test_reliable_qos ```

claraberendsen commented 3 months ago

:red_circle: ci.ros2.org build regressions (May 20th)

Out of space left on agent `linux-2xlarge-d29d1507`

Affected builds

nightly_linux_debug 3062
nightly_linux_release 3080
packaging_linux_rhel 1937
nightly_linux_humble_coverage #805

Action item: @claraberendsen - finish deploying the fix to ci.ros2.org

`ERROR: [ssh-agent] Could not register BouncyCastle on the remote agent.`

Affected builds

nightly_linux_aarch64_debug #2771 #2772

Previous job that failed seems like it could not clean itself properly. See reference

``` 02:01:54 Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from ip-10-0-2-85.ec2.internal/10.0.2.85:33064' is disconnected. 02:01:54 at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:216) 02:01:54 at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286) 02:01:54 at com.sun.proxy.$Proxy78.isAlive(Unknown Source) 02:01:54 at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1213) 02:01:54 at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1205) 02:01:54 at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:194) 02:01:54 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:144) 02:01:54 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91) 02:01:54 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) 02:01:54 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:806) 02:01:54 at hudson.model.Build$BuildExecution.build(Build.java:198) 02:01:54 at hudson.model.Build$BuildExecution.doRun(Build.java:163) 02:01:54 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:514) 02:01:54 at hudson.model.Run.execute(Run.java:1888) 02:01:54 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 02:01:54 at hudson.model.ResourceController.execute(ResourceController.java:99) 02:01:54 at hudson.model.Executor.run(Executor.java:432) 02:01:54 FATAL: Unable to delete script file /tmp/jenkins8655664486752392422.sh ```

Action item: @claraberendsen Rotate the agent so it comes back clean. :white_check_mark:

`nightly_win_deb` sros2 fatal exception

This seems to be the same issue that was happening last week see. Requires further investigation, it seems to be failing since May 16th.

`packaging_linux` untracked files of rti

Affected builds

packaging_linux #3445

Not sure where these files came from, I'll try to run a packaging job on the same machine and see if the files are still present.

``` 02:35:31 hudson.plugins.git.GitException: Command "git submodule update --init --recursive linux_docker_resources/rticonnextdds-src" returned status code 1: 02:35:31 stdout: 02:35:31 stderr: error: The following untracked working tree files would be overwritten by checkout: 02:35:31 README.md 02:35:31 openssl-1.1.1k-6.0.1.25-host-x64Linux.rtipkg 02:35:31 rti_connext_dds-6.0.1-pro-host-x64Linux.run.000 02:35:31 rti_connext_dds-6.0.1-pro-host-x64Linux.run.001 02:35:31 rti_connext_dds-6.0.1-pro-host-x64Linux.run.002 02:35:31 rti_connext_dds-6.0.1-pro-host-x64Linux.run.003 02:35:31 rti_connext_dds-6.0.1.25-pro-host-x64Linux.rtipkg.000 02:35:31 rti_connext_dds-6.0.1.25-pro-host-x64Linux.rtipkg.001 02:35:31 rti_connext_dds-6.0.1.25-pro-target-x64Linux4gcc7.3.0.rtipkg.000 02:35:31 rti_connext_dds-6.0.1.25-pro-target-x64Linux4gcc7.3.0.rtipkg.001 02:35:31 rti_security_plugins-6.0.1.25-host-x64Linux.rtipkg 02:35:31 rti_security_plugins-6.0.1.25-target-x64Linux4gcc7.3.0.rtipkg ```

claraberendsen commented 3 months ago

Nightly_win_deb sros2 failure investigation :female_detective:

First failure: https://ci.ros2.org/view/nightly/job/nightly_win_deb/3095/ Last failure: https://ci.ros2.org/view/nightly/job/nightly_win_deb/3099/

Error

04:05:34 test\sros2\commands\security\verbs\test_create_permission.py .           [ 65%]

04:05:34 Windows fatal exception: access violation
04:05:34 
04:05:34 Current thread 0x00002c94 (most recent call first):
04:05:34   File "C:\ci\ws\src\ros2\sros2\sros2\sros2\verb\generate_artifacts.py", line 44 in main

Possible cause: https://github.com/ros2/sros2/commit/fb22661623426fadcc60804e10a964664330d98e

claraberendsen commented 3 months ago

Coverage nightly for Jazzy and Rolling build regression :female_detective:

Affected builds

Seems like an issue when porting the coverage reports to the jenkins controller.

02:44:17 ERROR: ERROR: Failure to paint /home/jenkins-agent/workspace/nightly_linux_jazzy_coverage/ws/src/ros2/launch/launch/launch/event_handlers/on_include_launch_description.py to /var/lib/jenkins/jobs/nightly_linux_jazzy_coverage/cobertura
02:44:17 java.io.IOException: Failed to deserialize response to UserRequest:hudson.FilePath$WritePipe@2e47ea50: java.lang.SecurityException: Agent may not access a file path. See the system log for more details about the error ID '9da8cfdd-551a-403b-9459-8bee33963dd6' and https://www.jenkins.io/redirect/security-144 for more information.

nuclearsandwich commented 3 months ago

Seems like an issue when porting the coverage reports to the jenkins controller.

The issues logged have been present for years, quite likely, based on recent reports. Although those errors affect the visualization of the source files in coverage reports, they don't actually break the build.

Th last time this was being investigated, the exit status of colcon-test was non-zero and that was the "actual" hard failure.

clalancette commented 3 months ago

Th last time this was being investigated, the exit status of colcon-test was non-zero and that was the "actual" hard failure.

Yeah, we need to land https://github.com/ros2/ci/pull/771, which will switch this from a red build to a yellow one.

claraberendsen commented 3 months ago

Yeah, we need to land https://github.com/ros2/ci/pull/771, which will switch this from a red build to a yellow one.

Thanks, we will wait for that to land then. Sorry for the noise, it's been a while since I've done buildfarm work :laughing:

github-actions[bot] commented 3 months ago

May 21

Build regressions

``` nightly_linux_coverage|2392|2024-05-21 05:01:55.228000|Number of failed test exceeds the threshold|coverage analysis packaging_linux-rhel|1939|2024-05-21 07:42:15.319000|Packages fastrtps Failed|build underlay packages nightly_win_deb|3100|2024-05-21 04:00:01.299000|Number of failed test exceeds the threshold|test Rci__nightly-performance_ubuntu_noble_amd64|22|2024-05-21 00:53:40.532000|Packages performance_test Failed|build workspace in isolation and install gz_sensors-8-win|47|2024-05-20 13:06:10.840000|Packages gz-rendering8 Failed|colcon compilation without test for dependencies of gz-sensors8 sdformat-sdf14-win|51|2024-05-21 09:47:10.548000|Jenkins channel closing down (Agent disconnected)|clean up workspace ```

Test regressions

``` Excluding 'performance, connext, fastrtps-dynamic' jobs Issue #1 in nightly_linux-aarch64_repeated - Failing 93.33% in 14 out of 15 builds rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice projectroot.test_sequential_compression_writer (also in: "nightly_win_rep" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #2 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) Issue #3 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) Issue #6 in nightly_linux_repeated - Failing 31.25% in 5 out of 16 builds rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock projectroot.test.rclcpp.test_clock (also in: "nightly_win_rep" job) (also in: "nightly_win_rel" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) Issue #7 in gz_physics-ci-ign-physics2-homebrew-amd64 - Failing 100.0% in 4 out of 4 builds JointFeaturesFixture.JointDetach (also in: "gz_physics-ci-ign-physics5-homebrew-amd64" job) (also in: "gz_physics-ci-ign-physics2-homebrew-amd64" job) Issue #8 in gz_rendering-7-win - Failing 100.0% in 4 out of 4 builds MeshTest.NormalMapWithoutTexCoord (also in: "gz_rendering-main-win" job) (also in: "gz_rendering-8-win" job) (also in: "gz_rendering-7-win" job) Issue #9 in nightly_linux-rhel_repeated - Failing 25.0% in 4 out of 16 builds demo_nodes_cpp.TestExecutablesTutorial.test_processes_output (also in: "nightly_win_rep" job) (also in: "nightly_win_deb" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux_debug" job) (also in: "nightly_linux-rhel_repeated" job) Issue #10 in Jci__benchmark_ubuntu_noble_amd64 - Failing 100.0% in 3 out of 3 builds projectroot.benchmark_string_conversions (also in: "Rci__benchmark_ubuntu_noble_amd64" job) (also in: "Jci__benchmark_ubuntu_noble_amd64" job) Issue #11 in nightly_linux-aarch64_repeated - Failing 20.0% in 3 out of 15 builds rosbag2_transport.RecordIntegrationTestFixture.can_record_again_after_stop (also in: "nightly_linux-rhel_repeated" job) Issue #12 in nightly_linux-aarch64_repeated - Failing 20.0% in 3 out of 15 builds projectroot.test_record__rmw_fastrtps_cpp (also in: "nightly_win_rep" job) (also in: "nightly_linux-rhel_repeated" job) Issue #13 in nightly_linux-aarch64_repeated - Failing 20.0% in 3 out of 15 builds projectroot.test_record__rmw_cyclonedds_cpp Issue #14 in nightly_win_rel - Failing 21.43% in 3 out of 14 builds ros2cli.ros2cli.test.test_strategy.test_with_daemon_running (also in: "nightly_linux-rhel_debug" job) Issue #15 in Ici__nightly-debug_ubuntu_jammy_amd64 - Failing 40.0% in 2 out of 5 builds launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) (also in: "Hci__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) Issue #16 in gz_sim-main-win - Failing 50.0% in 2 out of 4 builds INTEGRATION_mesh_inertia_calculation.test_ran (also in: "gz_sim-8-win" job) Issue #17 in nightly_linux-aarch64_repeated - Failing 13.33% in 2 out of 15 builds test_rclcpp.TestNNodes.test_10_nodes (also in: "nightly_win_rep" job) (also in: "nightly_linux-rhel_repeated" job) Issue #18 in nightly_linux-aarch64_repeated - Failing 13.33% in 2 out of 15 builds rosbag2_py.test_transport_py.xunit.missing_result projectroot.test_transport_py Issue #19 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds rclcpp.TestBusyWaiting/EventsExecutor.test_spin Issue #20 in nightly_linux-aarch64_repeated - Failing 13.33% in 2 out of 15 builds test_rclcpp.TestNNodesAfterShutdown.test_10_nodes projectroot.test_n_nodes__rmw_fastrtps_cpp (also in: "nightly_win_rep" job) (also in: "nightly_linux-rhel_repeated" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #21 in Rci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 6.67% in 1 out of 15 builds sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes (also in: "nightly_win_rel" job) (also in: "nightly_win_deb" job) Issue #22 in Rci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 6.67% in 1 out of 15 builds ros2topic.ros2topic.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED) (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED) (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED) Issue #23 in Rci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 6.67% in 1 out of 15 builds ros2service.ros2service.test.test_cli.test_cli (known issue in: https://github.com/ros2/ros2cli/issues/861 - CLOSED) ros2doctor.ros2doctor.test.test_api.test_api Issue #24 in gz_rendering-ci-main-noble-amd64 - Failing 100.0% in 1 out of 1 builds INTEGRATION_load_unload_ogre_gl3plus.test_ran Issue #25 in nightly_linux-aarch64_debug - Failing 7.14% in 1 out of 14 builds launch_testing_examples.launch_testing_examples.check_node_launch_test.launch_testing_examples.check_node_launch_test (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-aarch64_repeated" job) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) Issue #26 in nightly_linux-aarch64_debug - Failing 7.14% in 1 out of 14 builds launch_testing_examples.launch_testing_examples.check_multiple_nodes_launch_test.launch_testing_examples.check_multiple_nodes_launch_test (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED) (also in: "nightly_linux_repeated" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED) (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED) (also in: "nightly_linux-aarch64_release" job) (known issue in: https://github.com/ros2/examples/issues/340 - CLOSED) Issue #27 in nightly_linux_iron_coverage - Failing 6.25% in 1 out of 16 builds test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully (also in: "nightly_win_rep" job) (also in: "nightly_linux-aarch64_release" job) Issue #28 in nightly_linux_iron_coverage - Failing 6.25% in 1 out of 16 builds test_security.TestSecurePublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_not_connecting_1 projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_not_connecting_0 projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_3 projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_2 projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_1 projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_0 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_not_connecting_1 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_not_connecting_0 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_3 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_2 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_1 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_0 (also in: "nightly_linux-aarch64_release" job) Issue #29 in nightly_win_rep - Failing 7.69% in 1 out of 13 builds rcl.TestPreInitTimer.test_timer_info_detect_overrun projectroot.test_test_lifecycle.py projectroot.test.test_timer__rmw_fastrtps_cpp projectroot.test.test_timer__rmw_cyclonedds_cpp lifecycle.TestLifecyclePubSub.test_talker_lifecycle ```

claraberendsen commented 3 months ago

:red_circle: Build regressions (May 21st)

ROS

Out of space left on agent `linux-2xlarge-98771619`

Affected builds

packaging_linux_rhel #1939

Action item: @claraberendsen - finish deploying the fix to ci.ros2.org. I was going to deploy today but need to patch buildfarm-deployment to deploy without mage.

Nightly win debug

Affected builds

nightly_win_deb #3100

Same issue as yesterday opening issue and reporting to the team. Issue opened https://github.com/ros2/sros2/issues/316

Coverage build regressions

Same as yesterday, waiting on PR to merge.

Rci_nightly_performance

Seems like an issue with dependencies not being imported correctly. Will open an issue. Seems to be happening for Rolling and Jazzy.

``` In file included from /tmp/ws2/src/performance_test/performance_test/src/experiment_configuration/qos_abstraction.cpp:15: 21:08:09 /tmp/ws2/src/performance_test/performance_test/src/experiment_configuration/qos_abstraction.hpp:61:3: error: ‘uint32_t’ does not name a type 21:08:09 61 | uint32_t history_depth; 21:08:09 | ^~~~~~~~ 21:08:09 /tmp/ws2/src/performance_test/performance_test/src/experiment_configuration/qos_abstraction.hpp:19:1: note: ‘uint32_t’ is defined in header ‘’; did you forget to ‘#include ’? 21:08:09 18 | #include 21:08:09 +++ |+#include 21:08:09 19 | 21:08:09 /tmp/ws2/src/performance_test/performance_test/src/experiment_configuration/qos_abstraction.hpp: In constructor ‘performance_test::QOSAbstraction::QOSAbstraction()’: 21:08:09 /tmp/ws2/src/performance_test/performance_test/src/experiment_configuration/qos_abstraction.hpp:32:5: error: class ‘performance_test::QOSAbstraction’ does not have any field named ‘history_depth’ 21:08:09 32 | history_depth(), 21:08:09 | ^~~~~~~~~~~~~ 21:08:09 /tmp/ws2/src/performance_test/performance_test/src/experiment_configuration/qos_abstraction.cpp: In function ‘std::ostream& performance_test::operator<<(std::ostream&, QOSAbstraction)’: 21:08:09 /tmp/ws2/src/performance_test/performance_test/src/experiment_configuration/qos_abstraction.cpp:59:34: error: ‘const struct performance_test::QOSAbstraction’ has no member named ‘history_depth’ 21:08:09 59 | " History depth: " << e.history_depth << 21:08:09 | ^~~~~~~~~~~~~ 21:08:09 gmake[2]: *** [CMakeFiles/perf_test.dir/build.make:150: CMakeFiles/perf_test.dir/src/experiment_configuration/qos_abstraction.cpp.o] Error 1 21:08:09 gmake[1]: *** [CMakeFiles/Makefile2:663: CMakeFiles/perf_test.dir/all] Error 2 ```

Gazebo

gz-sensors-win

Same issue reported to the team here https://github.com/gazebosim/gz-rendering/issues/1004

claraberendsen commented 3 months ago

:yellow_circle: Test regression `rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice` :female_detective:

Affected jobs

nightly_win_rep
nightly_linux_repeated
nightly_linux-rhel_repeated
nightly_linux-aarch64_repeated

Failing tests projectroot.test_sequential_compression_writer rosbag2_compression.SequentialCompressionWriterTest.open_succeeds_twice

Reference build: https://ci.ros2.org/job/nightly_linux-aarch64_repeated/2762 First build error: https://ci.ros2.org/job/nightly_linux-aarch64_repeated/2749/ This is happening consistent in nightlies. Possible cause for introduction: https://github.com/ros2/rosbag2/pull/1599 Error:

/home/jenkins-agent/workspace/nightly_linux-aarch64_repeated/ws/src/ros2/rosbag2/rosbag2_compression/test/rosbag2_compression/test_sequential_compression_writer.cpp:242
Expected: writer_->open(storage_options, {serialization_format_, serialization_format_}) doesn't throw an exception.
  Actual: it throws std::runtime_error with description "Bag directory already exists (/tmp/path_not_empty), can't overwrite existing bag

MichaelOrlov commented 3 months ago

@claraberendsen I've prepared a fix for the aforementioned CI regression. Please see https://github.com/ros2/rosbag2/pull/1667 for more information.

github-actions[bot] commented 3 months ago

May 22

Build regressions

``` nightly_linux_coverage|2396|2024-05-22 09:58:34.679000|Packages lttngpy Failed|build nightly_linux_coverage|2394|2024-05-22 05:37:37.343000|Number of failed test exceeds the threshold|coverage analysis nightly_win_deb|3101|2024-05-22 04:00:01.511000|Number of failed test exceeds the threshold|test Rci__nightly-performance_ubuntu_noble_amd64|23|2024-05-21 20:03:14.437000|Packages performance_test Failed|build workspace in isolation and install gz_launch-7-win|48|2024-05-21 23:06:17.901000|Packages gz-rendering8 Failed|colcon compilation without test for dependencies of gz-launch7 gz_sim-8-win|68|2024-05-21 17:06:12.250000|Packages gz-rendering8 Failed|colcon compilation without test for dependencies of gz-sim8 ```

Test regressions

``` Excluding 'performance, connext, fastrtps-dynamic' jobs Issue #1 in Rci__benchmark_ubuntu_noble_amd64 - Failing 100.0% in 12 out of 12 builds projectroot.benchmark_string_conversions (also in: "Jci__benchmark_ubuntu_noble_amd64" job) Issue #2 in nightly_win_rep - Failing 53.85% in 7 out of 13 builds projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) Issue #3 in nightly_linux_debug - Failing 46.67% in 7 out of 15 builds demo_nodes_cpp.TestExecutablesTutorial.test_processes_output (also in: "nightly_win_rep" job) (also in: "nightly_win_rel" job) (also in: "nightly_win_deb" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux_debug" job) (also in: "nightly_linux-rhel_repeated" job) Issue #4 in nightly_linux-rhel_repeated - Failing 37.5% in 6 out of 16 builds projectroot.test_showimage_cam2image__rmw_cyclonedds_cpp Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) Issue #7 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) Issue #8 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) Issue #9 in gz_rendering-7-win - Failing 100.0% in 4 out of 4 builds MeshTest.NormalMapWithoutTexCoord (also in: "gz_rendering-main-win" job) (also in: "gz_rendering-8-win" job) (also in: "gz_rendering-7-win" job) Issue #10 in nightly_linux-rhel_repeated - Failing 25.0% in 4 out of 16 builds rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock projectroot.test.rclcpp.test_clock (also in: "nightly_win_rep" job) (also in: "nightly_win_rel" job) (also in: "nightly_linux_repeated" job) Issue #11 in nightly_win_rep - Failing 30.77% in 4 out of 13 builds rclpy.test_service.xunit.missing_result Issue #12 in nightly_win_rep - Failing 30.77% in 4 out of 13 builds projectroot.test_timer (also in: "nightly_linux_iron_coverage" job) Issue #13 in nightly_win_rep - Failing 30.77% in 4 out of 13 builds rclpy.test_waitable.xunit.missing_result rclpy.test_wait_for_message.xunit.missing_result rclpy.test_timer.xunit.missing_result rclpy.test_time_source.xunit.missing_result rclpy.test_subscription.xunit.missing_result projectroot.test_waitable projectroot.test_wait_for_message projectroot.test_time_source Issue #14 in nightly_win_rep - Failing 30.77% in 4 out of 13 builds projectroot.test_subscription (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job) (also in: "Jci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job) (also in: "Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) Issue #15 in nightly_win_rep - Failing 30.77% in 4 out of 13 builds projectroot.test_service (also in: "nightly_win_deb" job) Issue #16 in nightly_linux-rhel_repeated - Failing 18.75% in 3 out of 16 builds test_rclcpp.TestNNodes.test_10_nodes (also in: "nightly_win_rep" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #17 in nightly_linux-rhel_repeated - Failing 18.75% in 3 out of 16 builds test_rclcpp.TestNNodesAfterShutdown.test_10_nodes projectroot.test_n_nodes__rmw_fastrtps_cpp (also in: "nightly_win_rep" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #18 in nightly_win_rep - Failing 23.08% in 3 out of 13 builds rclpy.test_node.xunit.missing_result projectroot.test_node (also in: "nightly_win_rel" job) Issue #19 in nightly_win_rep - Failing 23.08% in 3 out of 13 builds rclpy.test_service_introspection.xunit.missing_result projectroot.test_service_introspection projectroot.message_filters_test_latest_time_policy Issue #20 in Ici__nightly-debug_ubuntu_jammy_amd64 - Failing 40.0% in 2 out of 5 builds launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) (also in: "Hci__nightly-debug_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) Issue #21 in nightly_linux-rhel_repeated - Failing 12.5% in 2 out of 16 builds test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #22 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds robot_state_publisher.FixedJointTestAfterShutdown.test_exit_code Issue #23 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds rclpy.test_rate.xunit.missing_result projectroot.test_rate (also in: "nightly_win_rel" job) Issue #24 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds projectroot.test_publisher (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED) (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED) (also in: "Jci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED) (also in: "Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED) Issue #25 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds rclpy.test_rosout_subscription.xunit.missing_result rclpy.test_qos_overriding_options.xunit.missing_result rclpy.test_qos_event.xunit.missing_result rclpy.test_publisher.xunit.missing_result rclpy.test_parameter_service.xunit.missing_result rclpy.test_parameter_event_handler.xunit.missing_result rclpy.test_parameter_client.xunit.missing_result projectroot.test_rosout_subscription projectroot.test_qos_overriding_options projectroot.test_qos_event projectroot.test_parameter_service projectroot.test_parameter_event_handler projectroot.test_parameter_client Issue #26 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds message_filters.LatestTimePolicy.Leading Issue #27 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds projectroot.test_showimage_cam2image__rmw_fastrtps_cpp image_tools.TestExecutablesDemo.test_reliable_qos Issue #28 in gz_rendering-ci-main-noble-amd64 - Failing 100.0% in 1 out of 1 builds INTEGRATION_load_unload_ogre_gl3plus.test_ran Issue #29 in gz_sim-main-win - Failing 20.0% in 1 out of 5 builds INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) (also in: "gz_sim-ign-gazebo6-win" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) Issue #30 in nightly_linux-rhel_repeated - Failing 6.25% in 1 out of 16 builds launch_testing_examples.launch_testing_examples.check_node_launch_test.launch_testing_examples.check_node_launch_test (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-aarch64_debug" job) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) Issue #31 in nightly_linux_repeated - Failing 6.25% in 1 out of 16 builds rclcpp.TestAddCallbackGroupsToExecutorStable/SingleThreadedExecutor.subscriber_triggered_to_receive_message (also in: "nightly_linux-rhel_repeated" job) Issue #32 in nightly_linux_repeated - Failing 6.25% in 1 out of 16 builds launch_pytest.test.launch_pytest.tools.test_process.test_sync_process_tools[2-3] Issue #33 in nightly_win_rep - Failing 7.69% in 1 out of 13 builds rclcpp_lifecycle.TestDefaultStateMachine.wait_for_graph_change rclcpp_lifecycle.TestDefaultStateMachine.test_graph_topics rclcpp_lifecycle.TestDefaultStateMachine.test_graph_services_by_node rclcpp_lifecycle.TestDefaultStateMachine.test_graph_services rclcpp_lifecycle.TestDefaultStateMachine.test_callback_groups (also in: "nightly_win_deb" job) Issue #34 in nightly_win_rep - Failing 7.69% in 1 out of 13 builds projectroot.test_lifecycle_node (also in: "nightly_win_deb" job) (also in: "nightly_linux_iron_coverage" job) Issue #35 in nightly_win_rep - Failing 7.69% in 1 out of 13 builds ros2cli.ros2cli.test.test_strategy.test_with_no_daemon_running[1-3] rclcpp.TestExecutor.add_remove_node_thread_safe projectroot.test_two_links_fixed_joint_launch.py projectroot.test_tutorial_add_two_ints_server_add_two_ints_client__rmw_fastrtps_cpp launch_testing_ros.test.examples.talker_listener_launch_test.talker_listener_launch_test Issue #36 in nightly_win_rep - Failing 7.69% in 1 out of 13 builds launch_testing_ros.test.examples.check_node_launch_test.check_node_launch_test (also in: "nightly_linux-aarch64_repeated" job) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (also in: "Hci__nightly-connext_ubuntu_jammy_amd64" job) Issue #37 in nightly_win_rep - Failing 7.69% in 1 out of 13 builds demo_nodes_cpp.TestExecutablesTutorialAfterShutdown.test_last_process_exit_code (also in: "nightly_linux_repeated" job) ```

Crola1702 commented 3 months ago

Doc rosindex | Build regression | Keeping track :eyes:

Reference build: https://build.ros.org/job/doc_rosindex/2003/console (started failing again since build 1998)

Log output:

00:00:14.591 # BEGIN SECTION: Run Docker - rosindex
00:00:14.591 + export PODMAN_USERNS=keep-id
00:00:14.591 + docker run --rm --net=host --cidfile=/home/jenkins-agent/workspace/doc_rosindex/docker_rosindex.cid -v /home/jenkins-agent/workspace/doc_rosindex/repositories/rosindex:/tmp/doc_repository -v /home/jenkins-agent/workspace/doc_rosindex/upload_repository:/tmp/upload_repository -e REPO=/tmp/doc_repository -e SITE=/tmp/upload_repository rosindex
00:00:14.603 docker: container ID file found, make sure the other container isn't running or delete /home/jenkins-agent/workspace/doc_rosindex/docker_rosindex.cid.
00:00:14.603 See 'docker run --help'.
00:00:14.613 Build step 'Execute shell' marked build as failure
00:00:14.621 Build did not succeed and the project is configured to only push after a successful build, so no pushing will occur.
00:00:14.631 Finished: FAILURE

It's being solved in: https://github.com/ros-infrastructure/ros_buildfarm/pull/1047

Crola1702 commented 3 months ago

Ros2 nightlies `demo_nodes_cpp` | Test regression | Investigating :detective:

Reported in: https://github.com/ros2/demos/issues/693

There is a parent test regression with different variations on ConnextDDS: demo_nodes_cpp.TestExecutablesTutorial.test_processes_output

Failing test regressions:

projectroot.test_tutorial_parameter_events_async__rmw_connextdds
- Reference builds: linux debug 3064 - linux repeated 3459 - windows debug 3094
projectroot.test_tutorial_parameter_events__rmw_connextdds
- Reference builds: windows repeated 3362 - windows release 3053 - rhel repeated 1905

Log output (test_tutorial_parameter_events_async__rmw_connextdds):

``` FAIL: test_processes_output (demo_nodes_cpp.TestExecutablesTutorial.test_processes_output) Test all processes output against expectations. ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins-agent/workspace/nightly_linux_debug/ws/build/demo_nodes_cpp/test_parameter_events_async__rmw_connextdds_Debug.py", line 74, in test_processes_output proc_output.assertWaitFor( File "/home/jenkins-agent/workspace/nightly_linux_debug/ws/install/launch_testing/lib/python3.12/site-packages/launch_testing/io_handler.py", line 146, in assertWaitFor assert success, 'Waiting for output timed out' ^^^^^^^ AssertionError: Waiting for output timed out ```

Log output (test_tutorial_parameter_events__rmw_connextdds):

``` FAIL: test_processes_output (demo_nodes_cpp.TestExecutablesTutorial) Test all processes output against expectations. ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\ci\ws\build\demo_nodes_cpp\test_parameter_events__rmw_connextdds_Release.py", line 74, in test_processes_output proc_output.assertWaitFor( File "C:\ci\ws\install\Lib\site-packages\launch_testing\io_handler.py", line 146, in assertWaitFor assert success, 'Waiting for output timed out' AssertionError: Waiting for output timed out ```

Flakiness report (projectroot.test_tutorial_parameter_events_async__rmw_connextdds):

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
nightly_linux_debug	2024-05-22	2024-05-09	15	7	46.67
nightly_linux_repeated	2024-05-19	2024-05-08	16	2	12.5
nightly_win_deb	2024-05-15	2024-05-07	9	5	55.56

Flakiness report (projectroot.test_tutorial_parameter_events__rmw_connextdds):

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
nightly_win_rep	2024-05-22	2024-05-07	13	13	100.0
nightly_win_rel	2024-05-22	2024-05-22	14	1	7.14
nightly_linux-rhel_repeated	2024-05-21	2024-05-12	16	3	18.75
nightly_win_deb	2024-05-12	2024-05-09	9	2	22.22

I don't see any specific change that points to a reason why it started failing more in normal jobs, and not only repeated ones (package history).

projectroot.test_tutorial_parameter_events_async__rmw_connextdds

In linux debug started happening since March 31 build 3012
In linux repeated started happening since April 23 build 3433
In windows debug started happening since March 29 build 3047

projectroot.test_tutorial_parameter_events__rmw_connextdds:

In windows repeated this tests started happening since parallel testing (reference build) in Feb 25 and has been happening almost consistently since.
In windows release happened today only
In windows debug started happening since Feb 29 build 3018 (just after parallel testing)
In rhel repeated it started happening since March 8 build 1859

github-actions[bot] commented 3 months ago

May 23

Build regressions

``` nightly_linux_coverage|2400|2024-05-23 04:32:15.584000|Unknown Reason|Build Dockerfile nightly_linux_coverage|2399|2024-05-22 13:18:55.995000|Packages lttngpy Failed|build nightly_linux_debug|3065|2024-05-23 04:35:52.312000|Packages rviz_default_plugins Failed|build nightly_linux_humble_coverage|808|2024-05-23 04:49:19.249000|Early fail|No last section nightly_linux_iron_coverage|440|2024-05-23 04:59:39.265000|Packages test_tf2 Failed|build nightly_linux_repeated|3463|2024-05-23 06:10:29.243000|Packages launch_testing_examples ros2bag rosbag2 rosbag2_examples_py rosbag2_performance_benchmarking rosbag2_py rosbag2_tests rqt_bag rqt_bag_plugins Failed|ccache stats (after) nightly_win_deb|3102|2024-05-23 04:00:02.032000|Number of failed test exceeds the threshold|test nightly_win_rel|3054|2024-05-23 04:00:02.040000|Unknown Reason|test-result Ici__nightly-performance_ubuntu_jammy_amd64|99|2024-05-23 02:48:57.987000|Packages performance_test Failed|build workspace in isolation and install Rci__nightly-performance_ubuntu_noble_amd64|24|2024-05-22 20:54:54.315000|Packages performance_test Failed|build workspace in isolation and install Rci__nightly-release_ubuntu_noble_amd64|37|2024-05-22 18:45:15.106000|Number of failed test exceeds the threshold|run tests ```

Test regressions

``` Excluding 'performance, connext, fastrtps-dynamic' jobs Issue #1 in nightly_win_rep - Failing 100.0% in 13 out of 13 builds demo_nodes_cpp.TestExecutablesTutorial.test_processes_output (also in: "nightly_win_rel" job) (also in: "nightly_win_deb" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux_debug" job) (also in: "nightly_linux-rhel_repeated" job) Issue #2 in Rci__benchmark_ubuntu_noble_amd64 - Failing 100.0% in 12 out of 12 builds projectroot.benchmark_string_conversions (also in: "Jci__benchmark_ubuntu_noble_amd64" job) Issue #3 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 6 out of 6 builds projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 6 out of 6 builds projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 6 out of 6 builds projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 6 out of 6 builds rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) Issue #7 in nightly_win_rep - Failing 38.46% in 5 out of 13 builds test_rclcpp.test_timeout_subscriber.timeout_subscriber Issue #8 in gz_rendering-7-win - Failing 100.0% in 4 out of 4 builds MeshTest.NormalMapWithoutTexCoord (also in: "gz_rendering-main-win" job) (also in: "gz_rendering-7-win" job) Issue #9 in gz_rendering-ci-main-noble-amd64 - Failing 100.0% in 2 out of 2 builds INTEGRATION_load_unload_ogre_gl3plus.test_ran Issue #10 in nightly_linux-aarch64_release - Failing 12.5% in 2 out of 16 builds test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully (also in: "nightly_win_rep" job) (also in: "nightly_linux_iron_coverage" job) (also in: "nightly_linux-rhel_release" job) (also in: "nightly_linux-aarch64_release" job) Issue #11 in nightly_linux-aarch64_release - Failing 12.5% in 2 out of 16 builds test_security.TestSecurePublisherSubscriber.test_subscriber_terminates_in_a_finite_amount_of_time projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_not_connecting_1 projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_not_connecting_0 projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_3 projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_2 projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_1 projectroot.test_secure_publisher_subscriber__UnboundedSequences__rmw_cyclonedds_cpp__secure_comm_0 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_not_connecting_1 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_not_connecting_0 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_3 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_2 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_1 projectroot.test_secure_publisher_subscriber__Empty__rmw_cyclonedds_cpp__secure_comm_0 (also in: "nightly_linux_iron_coverage" job) (also in: "nightly_linux-rhel_release" job) (also in: "nightly_linux-aarch64_release" job) Issue #12 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds rosidl_typesupport_introspection_tests.ArraysMessageIntrospectionTest/0.CanWriteTypeErasedMessage projectroot.test_arrays_message_introspection Issue #13 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds projectroot.test_action_client Issue #14 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds projectroot.test_test_lifecycle.py lifecycle.TestLifecyclePubSub.test_talker_lifecycle Issue #15 in gz_sim-main-win - Failing 20.0% in 1 out of 5 builds INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) (also in: "gz_sim-ign-gazebo6-win" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) Issue #16 in nightly_linux-aarch64_repeated - Failing 6.67% in 1 out of 15 builds ros2bag.test.test_record.test_record Issue #17 in nightly_win_rep - Failing 7.69% in 1 out of 13 builds projectroot.test_record_all__rmw_cyclonedds_cpp (also in: "nightly_linux-aarch64_repeated" job) Issue #18 in nightly_win_rep - Failing 7.69% in 1 out of 13 builds test_communication.TestPublisherSubscriberAfterShutdown.test_processes_finished_gracefully[BoundedSequences] rcl_action.TestActionClientFixture.test_action_server_is_available projectroot.test_publisher_subscriber__rclcpp__rmw_fastrtps_cpp ```

Crola1702 commented 3 months ago

Ros linux nightlies | Build regression | Keeping track :eyes:

Some linux nightlies failed due to a "No Space Left" error in linux-2xlarge-f8f1b088 machine:

I'm cleaning up the space in the agent

github-actions[bot] commented 3 months ago

May 24

Build regressions

``` nightly_linux_coverage|2401|2024-05-24 05:46:22.585000|Number of failed test exceeds the threshold|coverage analysis packaging_linux-rhel|1943|2024-05-24 08:52:15.602000|Jenkins channel closing down (Agent disconnected)|build underlay packages Jci__nightly-performance_ubuntu_noble_amd64|10|2024-05-24 01:41:05.892000|Packages performance_test Failed|build workspace in isolation and install Rci__nightly-performance_ubuntu_noble_amd64|25|2024-05-24 02:20:44.991000|Packages performance_test Failed|build workspace in isolation and install ```

Test regressions

``` Excluding 'performance, connext, fastrtps-dynamic' jobs Issue #1 in nightly_linux_iron_coverage - Failing 46.67% in 7 out of 15 builds test_tracetools.test_tracetools.test.test_timer.TestTimer.test_all Issue #2 in nightly_linux_iron_coverage - Failing 46.67% in 7 out of 15 builds projectroot.test_timer (also in: "nightly_win_rep" job) Issue #3 in nightly_win_deb - Failing 87.5% in 7 out of 8 builds demo_nodes_cpp.TestExecutablesTutorial.test_processes_output (also in: "nightly_win_rep" job) (also in: "nightly_win_rel" job) (also in: "nightly_win_deb" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux_debug" job) (also in: "nightly_linux-rhel_repeated" job) Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) Issue #7 in nightly_win_rep - Failing 38.46% in 5 out of 13 builds test_rclcpp.test_timeout_subscriber.timeout_subscriber Issue #8 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_linux-aarch64_repeated" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) Issue #9 in gz_rendering-7-win - Failing 100.0% in 4 out of 4 builds MeshTest.NormalMapWithoutTexCoord (also in: "gz_rendering-main-win" job) (also in: "gz_rendering-8-win" job) (also in: "gz_rendering-7-win" job) Issue #10 in nightly_win_deb - Failing 50.0% in 4 out of 8 builds sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes (also in: "nightly_win_rel" job) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) Issue #11 in nightly_win_rep - Failing 30.77% in 4 out of 13 builds rclpy.test_time_source.xunit.missing_result projectroot.test_time_source Issue #12 in Jci__benchmark_ubuntu_noble_amd64 - Failing 100.0% in 3 out of 3 builds projectroot.benchmark_string_conversions (also in: "Rci__benchmark_ubuntu_noble_amd64" job) (also in: "Jci__benchmark_ubuntu_noble_amd64" job) Issue #13 in nightly_win_rep - Failing 23.08% in 3 out of 13 builds rosidl_typesupport_introspection_tests.MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage projectroot.test_multi_nested_message_introspection (also in: "nightly_linux_coverage" job) Issue #14 in nightly_win_rep - Failing 23.08% in 3 out of 13 builds projectroot.test_action_server Issue #15 in gz_rendering-ci-main-noble-amd64 - Failing 100.0% in 2 out of 2 builds INTEGRATION_load_unload_ogre_gl3plus.test_ran Issue #16 in gz_transport-12-win - Failing 50.0% in 2 out of 4 builds playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) (also in: "gz_transport-main-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) (also in: "gz_transport-ign-transport8-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) (also in: "gz_transport-13-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) (also in: "gz_transport-12-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) Issue #17 in nightly_linux-aarch64_repeated - Failing 13.33% in 2 out of 15 builds projectroot.test_record_all__rmw_cyclonedds_cpp (also in: "nightly_win_rep" job) Issue #18 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds projectroot.gtest_timeout_subscriber__rmw_cyclonedds_cpp Issue #19 in nightly_win_rep - Failing 15.38% in 2 out of 13 builds TestExecutors/StaticSingleThreadedExecutor.testSpinUntilSharedFutureComplete TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteWithTimeout TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteNoTimeout TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureCompleteInterrupted TestExecutors/StaticSingleThreadedExecutor.testSpinUntilFutureComplete TestExecutors/StaticSingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr TestExecutors/StaticSingleThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr TestExecutors/StaticSingleThreadedExecutor.testRaceConditionAddNode TestExecutors/StaticSingleThreadedExecutor.spin_some_max_duration TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) TestExecutors/StaticSingleThreadedExecutor.spinWithTimer TestExecutors/StaticSingleThreadedExecutor.spinWhileAlreadySpinning TestExecutors/StaticSingleThreadedExecutor.spinAll TestExecutors/StaticSingleThreadedExecutor.emptyExecutor TestExecutors/StaticSingleThreadedExecutor.detachOnDestruction TestExecutors/StaticSingleThreadedExecutor.addTemporaryNode TestExecutors/StaticSingleThreadedExecutor.addNodeTwoExecutors TestExecutors/MultiThreadedExecutor.testSpinUntilSharedFutureComplete TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteWithTimeout TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteNoTimeout TestExecutors/MultiThreadedExecutor.testSpinUntilFutureCompleteInterrupted TestExecutors/MultiThreadedExecutor.testSpinUntilFutureComplete TestExecutors/MultiThreadedExecutor.testSpinNodeUntilFutureCompleteNodePtr TestExecutors/MultiThreadedExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr TestExecutors/MultiThreadedExecutor.testRaceConditionAddNode TestExecutors/MultiThreadedExecutor.spin_some_max_duration TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) TestExecutors/MultiThreadedExecutor.spinWithTimer TestExecutors/MultiThreadedExecutor.spinWhileAlreadySpinning TestExecutors/MultiThreadedExecutor.spinAll TestExecutors/MultiThreadedExecutor.emptyExecutor TestExecutors/MultiThreadedExecutor.addTemporaryNode TestExecutors/MultiThreadedExecutor.addNodeTwoExecutors TestExecutors/EventsExecutor.testSpinUntilSharedFutureComplete TestExecutors/EventsExecutor.testSpinUntilFutureCompleteWithTimeout TestExecutors/EventsExecutor.testSpinUntilFutureCompleteNoTimeout TestExecutors/EventsExecutor.testSpinUntilFutureCompleteInterrupted TestExecutors/EventsExecutor.testSpinUntilFutureComplete TestExecutors/EventsExecutor.testSpinNodeUntilFutureCompleteNodePtr TestExecutors/EventsExecutor.testSpinNodeUntilFutureCompleteNodeBasePtr TestExecutors/EventsExecutor.testRaceConditionAddNode TestExecutors/EventsExecutor.spin_some_max_duration TestExecutors/EventsExecutor.spin_some TestExecutors/EventsExecutor.spinWithTimer TestExecutors/EventsExecutor.spinWhileAlreadySpinning TestExecutors/EventsExecutor.spinAll TestExecutors/EventsExecutor.emptyExecutor TestExecutors/EventsExecutor.detachOnDestruction TestExecutors/EventsExecutor.addTemporaryNode TestExecutors/EventsExecutor.addNodeTwoExecutors TestExecutors.testSpinWithNonDefaultContext TestExecutors.testSpinUntilFutureCompleteNodePtr TestExecutors.testSpinUntilFutureCompleteNodeBasePtr TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_all TestBusyWaiting/StaticSingleThreadedExecutor.test_spin TestBusyWaiting/SingleThreadedExecutor.test_spin_some TestBusyWaiting/SingleThreadedExecutor.test_spin_all TestBusyWaiting/SingleThreadedExecutor.test_spin TestBusyWaiting/MultiThreadedExecutor.test_spin_some TestBusyWaiting/MultiThreadedExecutor.test_spin_all TestBusyWaiting/MultiThreadedExecutor.test_spin TestBusyWaiting/EventsExecutor.test_spin_some TestBusyWaiting/EventsExecutor.test_spin_all TestBusyWaiting/EventsExecutor.test_spin Issue #20 in gz_sim-7-win - Failing 25.0% in 1 out of 4 builds INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) (also in: "gz_sim-main-win" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) (also in: "gz_sim-ign-gazebo6-win" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) (also in: "gz_sim-7-win" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) Issue #21 in nightly_linux-aarch64_repeated - Failing 6.67% in 1 out of 15 builds rcl.TestGraphFixture.test_graph_guard_condition_trigger_check (also in: "nightly_linux_repeated" job) Issue #22 in nightly_linux-aarch64_repeated - Failing 6.67% in 1 out of 15 builds test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp (also in: "nightly_linux_repeated" job) Issue #23 in nightly_linux-aarch64_repeated - Failing 6.67% in 1 out of 15 builds test_rmw_implementation.test_subscription__rmw_fastrtps_cpp.gtest.missing_result rcl.TestGetNodeNames.test_rcl_get_node_names_with_enclave projectroot.test_subscription__rmw_fastrtps_cpp projectroot.test.test_graph__rmw_cyclonedds_cpp projectroot.test.test_get_node_names__rmw_cyclonedds_cpp Issue #24 in nightly_win_deb - Failing 12.5% in 1 out of 8 builds robot_state_publisher.FixedJointTestAfterShutdown.test_exit_code (also in: "nightly_win_rep" job) Issue #25 in nightly_win_deb - Failing 12.5% in 1 out of 8 builds projectroot.test_two_links_fixed_joint_prefix_launch.py (also in: "nightly_win_rep" job) Issue #26 in nightly_win_deb - Failing 12.5% in 1 out of 8 builds projectroot.test_two_links_fixed_joint_launch.py (also in: "nightly_win_rep" job) Issue #27 in nightly_win_deb - Failing 12.5% in 1 out of 8 builds robot_state_publisher.MovingJointTestAfterShutdown.test_exit_code robot_state_publisher.ChangeFixedJointTestAfterShutdown.test_exit_code projectroot.test_two_links_moving_joint_launch.py projectroot.test_two_links_change_fixed_joint_launch.py projectroot.test_change_mimic_joint_launch.py Issue #28 in nightly_win_rep - Failing 7.69% in 1 out of 13 builds rcl_action.TestActionServer.test_action_clear_expired_goals projectroot.test_record_regex__rmw_cyclonedds_cpp ```

Crola1702 commented 3 months ago

Jazzy Release | Warnings | Investigating :detective:

Reference build:https://build.ros2.org/view/Jci/job/Jci__nightly-release_ubuntu_noble_amd64/15/

Warnings

atomic_base.h (stringop-overflow= warning)
benchmark.h (use-after-free warning)
OgreMesh.cpp (tringop-overflow= warning)
publisher_options.hpp (maybe-uninitialized warning)
relative_pointer.inl (array-bounds= warning)
stl_algo.h (array-bounds= warning)
stl_algobase.h (array-bounds= warning)
subscriber_options.hpp (maybe-uninitialized warning)

@clalancette these warnings are in different packages. Where should I report this?

github-actions[bot] commented 3 months ago

May 25

Build regressions

``` nightly_linux_repeated|3465|2024-05-25 07:05:08.847000|Packages rviz_default_plugins Failed|build packaging_linux|3452|2024-05-25 08:00:18.534000|Early fail|No last section Rci__nightly-release_ubuntu_noble_amd64|39|2024-05-24 19:07:00.457000|Number of failed test exceeds the threshold|run tests gz_sim-8-win|70|2024-05-25 09:50:09.983000|Jenkins channel closing down (Agent disconnected)|colcon compilation with tests for gz-sim8 ```

Test regressions

``` Excluding 'performance, connext, fastrtps-dynamic' jobs Issue #1 in nightly_win_rep - Failing 50.0% in 7 out of 14 builds projectroot.test.rclcpp.test_executor (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) Issue #2 in nightly_linux_debug - Failing 50.0% in 7 out of 14 builds demo_nodes_cpp.TestExecutablesTutorial.test_processes_output (also in: "nightly_win_rep" job) (also in: "nightly_win_rel" job) (also in: "nightly_win_deb" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux_debug" job) (also in: "nightly_linux-rhel_repeated" job) Issue #3 in nightly_win_rep - Failing 42.86% in 6 out of 14 builds test_rclcpp.test_timeout_subscriber.timeout_subscriber Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) Issue #6 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) Issue #7 in nightly_win_rep - Failing 35.71% in 5 out of 14 builds rclcpp.test_executor.gtest.missing_result (also in: "nightly_win_rel" job) Issue #8 in nightly_win_rep - Failing 35.71% in 5 out of 14 builds rclpy.test_time_source.xunit.missing_result rclpy.test_service.xunit.missing_result projectroot.test_time_source Issue #9 in nightly_win_rep - Failing 35.71% in 5 out of 14 builds projectroot.test_service (also in: "nightly_win_deb" job) Issue #10 in nightly_linux_coverage - Failing 55.56% in 5 out of 9 builds rosidl_typesupport_introspection_tests.MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage projectroot.test_multi_nested_message_introspection (also in: "nightly_win_rep" job) (also in: "nightly_linux_coverage" job) Issue #11 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 5 out of 5 builds projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) Issue #12 in gz_rendering-7-win - Failing 100.0% in 4 out of 4 builds MeshTest.NormalMapWithoutTexCoord (also in: "gz_rendering-main-win" job) (also in: "gz_rendering-8-win" job) (also in: "gz_rendering-7-win" job) Issue #13 in nightly_win_rep - Failing 28.57% in 4 out of 14 builds projectroot.test_timer (also in: "nightly_linux_iron_coverage" job) Issue #14 in nightly_win_rep - Failing 28.57% in 4 out of 14 builds projectroot.test_subscription (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job) (also in: "Jci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job) (also in: "Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) Issue #15 in nightly_win_rep - Failing 28.57% in 4 out of 14 builds rclpy.test_waitable.xunit.missing_result rclpy.test_wait_for_message.xunit.missing_result rclpy.test_timer.xunit.missing_result rclpy.test_subscription.xunit.missing_result rclpy.test_service_introspection.xunit.missing_result projectroot.test_waitable projectroot.test_wait_for_message projectroot.test_service_introspection Issue #16 in nightly_win_rep - Failing 28.57% in 4 out of 14 builds rclcpp.TestThreadSafeStorage.wait_waitable rclcpp.TestThreadSafeStorage.wait_timer rclcpp.TestThreadSafeStorage.wait_subscription rclcpp.TestThreadSafeStorage.wait_client_service rclcpp.TestThreadSafeStorage.add_remove_out_of_scope rclcpp.TestInterfaceTraits.has_node_base_interface projectroot.test.rclcpp.test_thread_safe_synchronization projectroot.test.rclcpp.test_logger_service projectroot.test.rclcpp.test_interface_traits (also in: "nightly_win_rel" job) Issue #17 in nightly_linux-rhel_repeated - Failing 25.0% in 4 out of 16 builds rclcpp.TestClockWakeup.multiple_threads_wait_on_one_clock projectroot.test.rclcpp.test_clock (also in: "nightly_win_rep" job) (also in: "nightly_win_rel" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) Issue #18 in nightly_linux-aarch64_repeated - Failing 20.0% in 3 out of 15 builds projectroot.test_record_all__rmw_cyclonedds_cpp (also in: "nightly_win_rep" job) Issue #19 in nightly_linux-rhel_repeated - Failing 18.75% in 3 out of 16 builds projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #20 in nightly_win_rel - Failing 23.08% in 3 out of 13 builds ros2cli.ros2cli.test.test_strategy.test_with_daemon_running (also in: "nightly_linux-rhel_debug" job) Issue #21 in nightly_win_rep - Failing 21.43% in 3 out of 14 builds test_security.TestSecurePublisherSubscriberAfterShutdown.test_processes_finished_gracefully (also in: "nightly_linux_iron_coverage" job) (also in: "nightly_linux-rhel_release" job) (also in: "nightly_linux-aarch64_release" job) Issue #22 in nightly_linux-rhel_repeated - Failing 18.75% in 3 out of 16 builds test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp (also in: "nightly_win_rep" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) (also in: "nightly_linux-aarch64_repeated" job) Issue #23 in nightly_win_rep - Failing 21.43% in 3 out of 14 builds rclpy.test_rate.xunit.missing_result projectroot.test_rate (also in: "nightly_win_rel" job) Issue #24 in nightly_win_rep - Failing 21.43% in 3 out of 14 builds projectroot.test_publisher (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED) (also in: "Rci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED) (also in: "Jci__nightly-fastrtps-dynamic_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED) (also in: "Ici__nightly-fastrtps-dynamic_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/system_tests/issues/496 - CLOSED) Issue #25 in nightly_win_rep - Failing 21.43% in 3 out of 14 builds rclpy.test_rosout_subscription.xunit.missing_result rclpy.test_qos_overriding_options.xunit.missing_result rclpy.test_qos_event.xunit.missing_result rclpy.test_publisher.xunit.missing_result rclpy.test_parameter_service.xunit.missing_result rclpy.test_parameter_event_handler.xunit.missing_result rclpy.test_parameter_client.xunit.missing_result projectroot.test_rosout_subscription projectroot.test_qos_overriding_options projectroot.test_qos_event projectroot.test_parameter_service projectroot.test_parameter_event_handler projectroot.test_parameter_client projectroot.test_default_state_machine Issue #26 in nightly_win_rep - Failing 21.43% in 3 out of 14 builds rclcpp.TestUtilities.wait_for_message_twice_one_sub rclcpp.TestUtilities.wait_for_message_indefinitely rclcpp.TestUtilities.wait_for_message rclcpp.TestTimeSource.reattach rclcpp.TestTimeSource.parameter_activation rclcpp.TestTimeSource.no_pre_jump_callback rclcpp.TestTimeSource.clock_sleep_until_with_ros_time_basic rclcpp.TestTimeSource.check_use_clock_thread_value rclcpp.TestTimeSource.check_clock_thread_status rclcpp.TestTimeSource.callbacks rclcpp.TestTimeSource.ROS_time_valid_wall_time rclcpp.TestTimeSource.ROS_time_valid_sim_time rclcpp.TestTimeSource.ROS_time_valid_attach_detach rclcpp.TestTimeSource.ROS_invalid_sim_time rclcpp.TestLoggerService.check_connect_get_logger_service projectroot.test.rclcpp.test_wait_for_message (also in: "nightly_win_rel" job) Issue #27 in nightly_win_rep - Failing 21.43% in 3 out of 14 builds rclcpp.TestTimeSource.clock rclcpp.TestTimeSource.callback_handler_erasure projectroot.test.rclcpp.test_time_source (also in: "nightly_win_rel" job) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) Issue #28 in nightly_win_rep - Failing 21.43% in 3 out of 14 builds projectroot.test.rclcpp.test_subscription_topic_statistics (also in: "nightly_win_rel" job) (also in: "Rci__nightly-connext_ubuntu_noble_amd64" job) (also in: "Jci__nightly-connext_ubuntu_noble_amd64" job) Issue #29 in nightly_win_rep - Failing 21.43% in 3 out of 14 builds rclcpp.TestWaitSet.add_remove_wait rclcpp.TestWaitSet.add_guard_condition_to_two_different_wait_set rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testTimer2CancelledWithExecutorSpin rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testTimer1CancelledWithExecutorSpin rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testHeadTimerCancelThenResetBehavior rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBothTimerCancelThenResetT2Behavior rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBothTimerCancelThenResetT1Behavior rclcpp.TestTimerCancelBehavior/SingleThreadedExecutor.testBackTimerCancelThenResetBehavior rclcpp.TestThreadSafeStorage.iterables_construct_destruct rclcpp.TestThreadSafeStorage.default_construct_destruct rclcpp.TestThreadSafeStorage.add_remove_nullptr rclcpp.TestThreadSafeStorage.add_remove_dynamically rclcpp.TestSubscriptionTopicStatisticsFixture.test_invalid_publish_period rclcpp.TestSubscriptionOptions.topic_statistics_options_node_default_mode rclcpp.TestStoragePolicyCommon.rcl_wait_set_resize_error rclcpp.TestStoragePolicyCommon.rcl_wait_set_fini_error rclcpp.TestStoragePolicyCommon.rcl_wait_set_clear_error rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_timer_error rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_subscription_error rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_service_error rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_guard_condition_error rclcpp.TestStoragePolicyCommon.rcl_wait_set_add_client_error rclcpp.TestStoragePolicyCommon.add_waitable_error rclcpp.TestStaticStorage.wait_waitable rclcpp.TestStaticStorage.wait_timer rclcpp.TestStaticStorage.wait_subscription rclcpp.TestStaticStorage.wait_guard_condition rclcpp.TestStaticStorage.wait_client_service rclcpp.TestStaticStorage.iterables_construct_destruct rclcpp.TestStaticStorage.fixed_storage_needs_pruning rclcpp.TestStaticSingleThreadedExecutor.add_callback_group_trigger_guard_failed rclcpp.TestRosoutSubscription.test_rosoutsubscription_parent_log rclcpp.TestRosoutSubscription.test_rosoutsubscription_node_rosout_disabled rclcpp.TestRosoutSubscription.test_rosoutsubscription_getchild_hierarchy rclcpp.TestRosoutSubscription.test_rosoutsubscription_getchild rclcpp.TestRosoutSubscription.test_rosoutsubscription_child_log rclcpp.TestMultiThreadedExecutor.timer_over_take rclcpp.TestIntraprocessExecutors/SingleThreadedExecutor.testIntraprocessRetrigger rclcpp.TestIntraProcessWaitable.test_that_waitable_stays_ready_after_second_wait rclcpp.TestGraphListener.construction_and_destruction rclcpp.TestExecutors/SingleThreadedExecutor.detachOnDestruction rclcpp.TestExecutorNotifyWaitable.add_remove_guard_conditions rclcpp.TestExecutorEntitiesCollector.add_remove_node rclcpp.TestEventsExecutor.run_pub_sub rclcpp.TestDynamicStorage.wait_waitable rclcpp.TestDynamicStorage.wait_timer rclcpp.TestDynamicStorage.wait_subscription rclcpp.TestDynamicStorage.wait_client_service rclcpp.TestDynamicStorage.iterables_construct_destruct rclcpp.TestDynamicStorage.default_construct_destruct rclcpp.TestDynamicStorage.add_remove_out_of_scope rclcpp.TestDynamicStorage.add_remove_nullptr rclcpp.TestDynamicStorage.add_remove_dynamically rclcpp.TestContentFilterSubscription.set_content_filter_error rclcpp.TestContentFilterSubscription.set_content_filter rclcpp.TestContentFilterSubscription.is_cft_enabled rclcpp.TestContentFilterSubscription.get_content_filter_error rclcpp.TestContentFilterSubscription.get_content_filter rclcpp.TestContentFilterSubscription.content_filter_reset rclcpp.TestContentFilterSubscription.content_filter_get_later rclcpp.TestContentFilterSubscription.content_filter_get_begin rclcpp.TestCallbackGroup.valid_callback_group rclcpp.TestCallbackGroup.invalid_callback_group projectroot.test.rclcpp.test_wait_set projectroot.test.rclcpp.test_subscription_options projectroot.test.rclcpp.test_subscription_content_filter__rmw_fastrtps_cpp projectroot.test.rclcpp.test_storage_policy_common projectroot.test.rclcpp.test_static_storage projectroot.test.rclcpp.test_static_single_threaded_executor projectroot.test.rclcpp.test_rosout_subscription projectroot.test.rclcpp.test_multi_threaded_executor projectroot.test.rclcpp.test_intra_process_waitable projectroot.test.rclcpp.test_graph_listener projectroot.test.rclcpp.test_executors_timer_cancel_behavior projectroot.test.rclcpp.test_executors_intraprocess projectroot.test.rclcpp.test_executors_callback_group_behavior projectroot.test.rclcpp.test_executor_notify_waitable projectroot.test.rclcpp.test_events_executor projectroot.test.rclcpp.test_entities_collector projectroot.test.rclcpp.test_dynamic_storage (also in: "nightly_win_rel" job) Issue #30 in Jci__benchmark_ubuntu_noble_amd64 - Failing 100.0% in 2 out of 2 builds projectroot.benchmark_string_conversions (also in: "Rci__benchmark_ubuntu_noble_amd64" job) (also in: "Jci__benchmark_ubuntu_noble_amd64" job) Issue #31 in gz_rendering-ci-main-noble-amd64 - Failing 100.0% in 2 out of 2 builds INTEGRATION_load_unload_ogre_gl3plus.test_ran Issue #32 in gz_transport-12-win - Failing 50.0% in 2 out of 4 builds playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) (also in: "gz_transport-main-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) (also in: "gz_transport-ign-transport8-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) (also in: "gz_transport-12-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) Issue #33 in nightly_win_rep - Failing 14.29% in 2 out of 14 builds test_cli_remapping.TestCLIRemappingAfterShutdown.test_processes_finished_gracefully[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe] rcl_lifecycle.TestDefaultStateMachine.default_sequence Issue #34 in nightly_win_rep - Failing 14.29% in 2 out of 14 builds rclcpp.TestClockSleep.sleep_for_shutdown_interrupt projectroot.test_secure_publisher_subscriber__Empty__rmw_fastrtps_cpp__non_secure_comm_2 Issue #35 in nightly_win_rep - Failing 14.29% in 2 out of 14 builds rcl_lifecycle.TestRclLifecycle.state_machine rcl_lifecycle.TestMultipleInstances.default_sequence_error_unresolved projectroot.test_rcl_lifecycle projectroot.test_multiple_instances Issue #36 in nightly_win_rep - Failing 14.29% in 2 out of 14 builds projectroot.test_cli_remapping (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) Issue #37 in nightly_win_rep - Failing 14.29% in 2 out of 14 builds projectroot.test_arrays_service_introspection Issue #38 in nightly_win_rep - Failing 14.29% in 2 out of 14 builds rclcpp.TestTimeSource.detachUnattached rclcpp.PerTimerType/TestTimer.test_simple_cancel/wall_timer (4-byte object <00-00 00-00>) projectroot.test.rclcpp.test_timer (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED) projectroot.test.rclcpp.test_time (known issue in: https://github.com/ros2/rclcpp/issues/2436 - CLOSED) (also in: "nightly_win_rel" job) Issue #39 in gz_sim-ci-ign-gazebo6-homebrew-amd64 - Failing 25.0% in 1 out of 4 builds INTEGRATION_triggered_publisher.test_ran Issue #40 in gz_sim-7-win - Failing 25.0% in 1 out of 4 builds INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) (also in: "gz_sim-main-win" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) (also in: "gz_sim-ign-gazebo6-win" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) (also in: "gz_sim-7-win" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) Issue #41 in nightly_linux-aarch64_repeated - Failing 6.67% in 1 out of 15 builds ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[1-3] (also in: "nightly_win_rep" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) Issue #42 in nightly_win_deb - Failing 12.5% in 1 out of 8 builds test_rmw_implementation.test_subscription__rmw_fastrtps_cpp.gtest.missing_result projectroot.test_subscription__rmw_fastrtps_cpp (also in: "nightly_linux-aarch64_repeated" job) Issue #43 in nightly_win_deb - Failing 12.5% in 1 out of 8 builds test_rmw_implementation.test_wait_set__rmw_fastrtps_cpp.gtest.missing_result test_rmw_implementation.test_unique_identifiers__rmw_fastrtps_cpp.gtest.missing_result test_rmw_implementation.test_service__rmw_fastrtps_cpp.gtest.missing_result test_rmw_implementation.test_graph_api__rmw_fastrtps_cpp.gtest.missing_result test_rmw_implementation.test_event__rmw_fastrtps_cpp.gtest.missing_result test_rmw_implementation.test_client__rmw_fastrtps_cpp.gtest.missing_result test_rmw_implementation.TestDurationInfinite.create_publisher projectroot.test_wait_set__rmw_fastrtps_cpp projectroot.test_unique_identifiers__rmw_fastrtps_cpp projectroot.test_service__rmw_fastrtps_cpp projectroot.test_graph_api__rmw_fastrtps_cpp projectroot.test_event__rmw_fastrtps_cpp projectroot.test_duration_infinite__rmw_fastrtps_cpp projectroot.test_client__rmw_fastrtps_cpp Issue #44 in nightly_win_rep - Failing 7.14% in 1 out of 14 builds test_rclcpp.gtest_executor__rmw_fastrtps_cpp.gtest.missing_result test_launch_ros.test.test_launch_ros.utilities.test_track_node_names.test_launch_nodes_with_different_names[2-3] test_cli_remapping.TestCLIRemapping.test_service_replacement[C:/ci/ws/build/test_cli_remapping/Release/name_maker_rclcpp.exe] rosidl_typesupport_introspection_tests.ArraysServiceIntrospectionTest/0.CanReadTypeErasedResponseMessage projectroot.test_client_scope_consistency_cpp__rmw_fastrtps_cpp projectroot.gtest_executor__rmw_fastrtps_cpp ```

github-actions[bot] commented 3 months ago

May 26

Build regressions

``` nightly_linux_debug|3068|2024-05-26 04:20:47.078000|Packages rosbag2_tests Failed|build nightly_linux_humble_coverage|811|2024-05-26 04:36:09.935000|Early fail|No last section nightly_linux_iron_coverage|443|2024-05-26 04:46:29.802000|Packages rosbag2_compression Failed|build nightly_linux_release|3086|2024-05-26 05:18:37.128000|Packages action_tutorials_cpp action_tutorials_py camera_calibration_parsers camera_info_manager common_interfaces composition demo_nodes_cpp demo_nodes_cpp_native demo_nodes_py diagnostic_msgs dummy_map_server dummy_robot_bringup dummy_sensors examples_rclcpp_async_client examples_rclcpp_cbg_executor examples_rclcpp_minimal_action_client examples_rclcpp_minimal_action_server examples_rclcpp_minimal_client examples_rclcpp_minimal_composition examples_rclcpp_minimal_publisher examples_rclcpp_minimal_service examples_rclcpp_minimal_subscriber examples_rclcpp_minimal_timer examples_rclcpp_multithreaded_executor examples_rclcpp_wait_set examples_rclpy_executors examples_rclpy_guard_conditions examples_rclpy_minimal_action_client examples_rclpy_minimal_action_server examples_rclpy_minimal_client examples_rclpy_minimal_publisher examples_rclpy_minimal_service examples_rclpy_minimal_subscriber examples_rclpy_pointcloud_publisher examples_tf2_py geometry2 image_common image_tools image_transport interactive_markers intra_process_demo laser_geometry launch_ros launch_testing_examples launch_testing_ros libstatistics_collector lifecycle lifecycle_py logging_demo map_msgs message_filters nav_msgs pendulum_control point_cloud_transport point_cloud_transport_py quality_of_service_demo_cpp quality_of_service_demo_py rcl rcl_action rcl_lifecycle rclcpp rclcpp_action rclcpp_components rclcpp_lifecycle rclpy rmw_cyclonedds_cpp rmw_fastrtps_cpp rmw_implementation robot_state_publisher ros2action ros2bag ros2cli ros2cli_common_extensions ros2component ros2doctor ros2interface ros2launch ros2lifecycle ros2lifecycle_test_fixtures ros2multicast ros2node ros2param ros2pkg ros2run ros2service ros2test ros2topic ros2trace ros_testing rosbag2 rosbag2_compression rosbag2_compression_zstd rosbag2_cpp rosbag2_examples_cpp rosbag2_examples_py rosbag2_performance_benchmarking rosbag2_py rosbag2_storage rosbag2_storage_default_plugins rosbag2_storage_mcap rosbag2_storage_sqlite3 rosbag2_test_common rosbag2_tests rosbag2_transport rosidl_runtime_py rosidl_typesupport_tests rqt rqt_action rqt_bag rqt_bag_plugins rqt_console rqt_graph rqt_gui rqt_gui_cpp rqt_gui_py rqt_msg rqt_plot rqt_publisher rqt_py_common rqt_py_console rqt_reconfigure rqt_service_caller rqt_shell rqt_srv rqt_topic rviz2 rviz_common rviz_default_plugins rviz_visual_testing_framework sensor_msgs sensor_msgs_py shape_msgs sros2 sros2_cmake stereo_msgs test_cli test_cli_remapping test_communication test_launch_ros test_quality_of_service test_rclcpp test_rmw_implementation test_ros2trace test_security test_tf2 test_tracetools test_tracetools_launch tf2 tf2_bullet tf2_eigen tf2_eigen_kdl tf2_geometry_msgs tf2_kdl tf2_msgs tf2_py tf2_ros tf2_ros_py tf2_sensor_msgs tf2_tools tlsf_cpp topic_monitor topic_statistics_demo tracetools_launch tracetools_test trajectory_msgs turtlesim visualization_msgs Failed|ccache stats (after) packaging_linux|3453|2024-05-26 06:23:54.238000|Early fail|No last section nightly_linux-rhel_repeated|1910|2024-05-26 04:00:01.488000|Early fail|No last section nightly_win_rep|3366|2024-05-26 04:00:01.510000|Number of failed test exceeds the threshold|test Rci__nightly-performance_ubuntu_noble_amd64|27|2024-05-25 23:34:20.576000|Packages performance_test Failed|build workspace in isolation and install ```

Test regressions

``` Excluding 'performance, connext, fastrtps-dynamic' jobs Issue #1 in nightly_win_deb - Failing 87.5% in 7 out of 8 builds demo_nodes_cpp.TestExecutablesTutorial.test_processes_output (also in: "nightly_win_rep" job) (also in: "nightly_win_rel" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux_debug" job) (also in: "nightly_linux-rhel_repeated" job) Issue #2 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 6 out of 6 builds projectroot.test_lifecycle_service_client (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "nightly_win_deb" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) Issue #3 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 6 out of 6 builds projectroot.test_guard_condition (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) Issue #4 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 6 out of 6 builds projectroot.test.rclcpp.test_executors (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rep" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "nightly_win_rel" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) (also in: "Ici__nightly-connext_ubuntu_jammy_amd64" job) (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) Issue #5 in Jci__nightly-cyclonedds_ubuntu_noble_amd64 - Failing 100.0% in 6 out of 6 builds rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclpy.rclpy.test.test_guard_condition.TestGuardCondition.test_double_trigger (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) rclcpp_lifecycle.TestLifecycleServiceClientRCLErrors.call_services_rcl_errors (known issue in: https://github.com/ros2/rclcpp/issues/2472 - OPEN) rclcpp.TestExecutors/StaticSingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/SingleThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/SingleThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestExecutors/MultiThreadedExecutor.spin_some_max_duration rclcpp.TestExecutors/MultiThreadedExecutor.spin_some (known issue in: https://github.com/ros2/rclcpp/issues/2502 - OPEN) rclcpp.TestBusyWaiting/StaticSingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/SingleThreadedExecutor.test_spin_all rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_some rclcpp.TestBusyWaiting/MultiThreadedExecutor.test_spin_all (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) (known issue in: https://github.com/ros2/rclpy/issues/1266 - OPEN) (also in: "Jci__nightly-cyclonedds_ubuntu_noble_amd64" job) Issue #6 in nightly_linux_coverage - Failing 60.0% in 6 out of 10 builds rosidl_typesupport_introspection_tests.MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage projectroot.test_multi_nested_message_introspection (also in: "nightly_win_rep" job) Issue #7 in nightly_win_deb - Failing 50.0% in 4 out of 8 builds sros2.test.sros2.commands.security.verbs.test_generate_policy_no_nodes.test_generate_policy_no_nodes (also in: "nightly_win_rel" job) (also in: "Rci__nightly-cyclonedds_ubuntu_noble_amd64" job) Issue #8 in Ici__nightly-debug_ubuntu_jammy_amd64 - Failing 60.0% in 3 out of 5 builds launch_testing_ros.test.examples.wait_for_topic_launch_test.wait_for_topic_launch_test (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) (also in: "nightly_linux_iron_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) (also in: "nightly_linux_coverage" job) (known issue in: https://github.com/ros2/launch_ros/issues/304 - CLOSED) Issue #9 in gz_sim-main-win - Failing 60.0% in 3 out of 5 builds INTEGRATION_mesh_inertia_calculation.test_ran Issue #10 in nightly_linux_coverage - Failing 30.0% in 3 out of 10 builds ros2topic.ros2topic.test.test_use_sim_time.test_use_sim_time (known issue in: https://github.com/ros2/ros2cli/issues/772 - CLOSED) Issue #11 in Jci__benchmark_ubuntu_noble_amd64 - Failing 100.0% in 2 out of 2 builds projectroot.benchmark_string_conversions (also in: "Rci__benchmark_ubuntu_noble_amd64" job) (also in: "Jci__benchmark_ubuntu_noble_amd64" job) Issue #12 in gz_rendering-ci-main-noble-amd64 - Failing 100.0% in 2 out of 2 builds INTEGRATION_load_unload_ogre_gl3plus.test_ran Issue #13 in nightly_linux-aarch64_repeated - Failing 13.33% in 2 out of 15 builds test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_cpp projectroot.test_client_scope_cpp__rmw_cyclonedds_cpp (also in: "nightly_linux_repeated" job) Issue #14 in nightly_win_deb - Failing 25.0% in 2 out of 8 builds test_rclcpp.test_multithreaded.multi_access_publisher Issue #15 in gz_gui-ci-ign-gui6-focal-amd64 - Failing 20.0% in 1 out of 5 builds UNIT_Screenshot_TEST.test_ran (also in: "gz_gui-ci-gz-gui7-homebrew-amd64" job) Issue #16 in gz_gui-ci-ign-gui6-focal-amd64 - Failing 20.0% in 1 out of 5 builds UNIT_Teleop_TEST.test_ran (known issue in: https://github.com/gazebosim/gz-gui/issues/503 - CLOSED) UNIT_ShutdownButton_TEST.test_ran UNIT_Publisher_TEST.test_ran Issue #17 in gz_rendering-8-win - Failing 100.0% in 1 out of 1 builds MeshTest.NormalMapWithoutTexCoord (also in: "gz_rendering-main-win" job) (also in: "gz_rendering-8-win" job) (also in: "gz_rendering-7-win" job) Issue #18 in gz_sim-7-win - Failing 25.0% in 1 out of 4 builds INTEGRATION_save_world.test_ran (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) (also in: "gz_sim-main-win" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) (also in: "gz_sim-ign-gazebo6-win" job) (known issue in: https://github.com/gazebosim/gz-sim/issues/2177 - CLOSED) Issue #19 in gz_sim-ci-ign-gazebo6-homebrew-amd64 - Failing 25.0% in 1 out of 4 builds INTEGRATION_triggered_publisher.test_ran Issue #20 in gz_transport-12-win - Failing 33.33% in 1 out of 3 builds playback.ReplayStep (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) (also in: "gz_transport-main-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) (also in: "gz_transport-ign-transport8-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) (also in: "gz_transport-12-win" job) (known issue in: https://github.com/gazebosim/gz-transport/issues/195 - CLOSED | https://github.com/gazebosim/gz-transport/issues/195 - OPEN) Issue #21 in nightly_linux-aarch64_repeated - Failing 6.67% in 1 out of 15 builds test_rclcpp.TestTwoExecutablesAfterShutdown.test_client_scope_consistency_cpp (also in: "nightly_win_rep" job) (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) Issue #22 in nightly_linux-aarch64_repeated - Failing 6.67% in 1 out of 15 builds ros2cli.ros2cli.test.test_strategy.test_with_daemon_spawn[3-3] (also in: "nightly_linux_repeated" job) Issue #23 in nightly_linux-aarch64_repeated - Failing 6.67% in 1 out of 15 builds projectroot.test_client_scope_consistency_cpp__rmw_cyclonedds_cpp (also in: "nightly_linux_repeated" job) (also in: "nightly_linux-rhel_repeated" job) Issue #24 in nightly_win_rel - Failing 7.69% in 1 out of 13 builds projectroot.test_action_server (also in: "nightly_win_rep" job) Issue #25 in nightly_win_rel - Failing 7.69% in 1 out of 13 builds rcl_action.TestActionServerInitFini.test_action_server_init_fini rcl_action.TestActionGraphFixture__rmw_fastrtps_cpp.test_action_get_client_names_and_types_by_node rcl_action.TestActionCommunication__rmw_fastrtps_cpp.test_valid_goal_comm rcl_action.TestActionClientWait.test_wait_set_add_action_client rcl_action.TestActionClientServerInteraction__rmw_fastrtps_cpp.test_interaction rcl_action.TestActionClientBaseFixture.test_action_client_init_fini projectroot.test_wait projectroot.test_graph__rmw_fastrtps_cpp projectroot.test_action_interaction__rmw_fastrtps_cpp projectroot.test_action_communication__rmw_fastrtps_cpp Issue #26 in nightly_win_rel - Failing 7.69% in 1 out of 13 builds projectroot.test_action_client (also in: "nightly_win_rep" job) ```

osrf / buildfarm-tools