:man_farmer: Flaky issues with `launch_testing_ros`: `terminate called without an active exception`

[Crola1702]

Bug report

Required Info:

• Operating System:
  • Ubuntu Jammy
• Installation type:
  • Source, buildfarm
• Version or commit hash:
  • Rolling, master
• DDS implementation:
  • Fast-RTPS, CycloneDDS

Steps to reproduce issue

1. Run Linux nightlies multiple times
2. Wait for failures

Expected behavior

Build successfully

Actual behavior

Failing in launch_testing_ros with this message:

pure virtual method called
terminate called without an active exception

Log output:

``` test/examples/wait_for_topic_launch_test.py::wait_for_topic_launch_test Warning: Exception in thread Thread-8 (spin) Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 946, in run self._target(*self._args, **self._kwargs) File "/home/jenkins-agent/workspace/nightly_linux-aarch64_release/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 279, in spin self.spin_once() File "/home/jenkins-agent/workspace/nightly_linux-aarch64_release/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 717, in spin_once handler, entity, node = self.wait_for_ready_callbacks(timeout_sec=timeout_sec) File "/home/jenkins-agent/workspace/nightly_linux-aarch64_release/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 703, in wait_for_ready_callbacks return next(self._cb_iter) File "/home/jenkins-agent/workspace/nightly_linux-aarch64_release/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 605, in _wait_for_ready_callbacks raise ExternalShutdownException() rclpy.executors.ExternalShutdownException -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html pure virtual method called terminate called without an active exception --- Failed <<< launch_testing_ros [21.8s, exited with code -6] ```

Additional information

Reference builds on nightly_linux-aarch64: 2045, 2042

Reference build on Nightly Linux release: 2396

This issue might be related to #304

[Crola1702]

It appeared again today in Linux repeated job: 2778

[Crola1702]

It appeared again today on Linux repeated job: 2783

This one only has terminate called without an active exception

[Crola1702]

It appeared today on Linux repeated job: 2791, and Linux RHEL debug 1275

Both of them only have terminate called without an active exception

[Crola1702]

It appeared today on Linux aarch64 repeated job: 2125

Log output:

pure virtual method called
terminate called without an active exception

[Crola1702]

Now It appeared on Linux aarch64 debug: 2144

Log output:

terminate called without an active exception

[Crola1702]

Happened today in Linux aarch64 debug: 2460

Log output:

terminate called without an active exception\
...
Failed   <<< launch_testing_ros [22.3s, exited with code -6]

[Crola1702]

Today it happened in Rolling Release: 264

Log output:

test/examples/wait_for_topic_launch_test.py: 2 warnings
  Warning: There is no current event loop

test/examples/wait_for_topic_launch_test.py::wait_for_topic_launch_test
  Warning: Exception in thread Thread-8 (spin)

  Traceback (most recent call last):
    File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
      self.run()
    File "/usr/lib/python3.10/threading.py", line 953, in run
      self._target(*self._args, **self._kwargs)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 280, in spin
      self.spin_once()
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 717, in spin_once
      handler, entity, node = self.wait_for_ready_callbacks(timeout_sec=timeout_sec)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 703, in wait_for_ready_callbacks
      return next(self._cb_iter)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 604, in _wait_for_ready_callbacks
      raise ExternalShutdownException()
  rclpy.executors.ExternalShutdownException

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
pure virtual method called
terminate called without an active exception
...
Failed   <<< launch_testing_ros [20.5s, exited with code -6]

[Crola1702]

Yesterday it happened on Rolling Release: 267

test/examples/wait_for_topic_launch_test.py::wait_for_topic_launch_test
  Warning: Exception in thread Thread-8 (spin)

  Traceback (most recent call last):
    File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
      self.run()
    File "/usr/lib/python3.10/threading.py", line 953, in run
      self._target(*self._args, **self._kwargs)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 280, in spin
      self.spin_once()
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 717, in spin_once
      handler, entity, node = self.wait_for_ready_callbacks(timeout_sec=timeout_sec)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 703, in wait_for_ready_callbacks
      return next(self._cb_iter)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 604, in _wait_for_ready_callbacks
      raise ExternalShutdownException()
  rclpy.executors.ExternalShutdownException

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---- generated xml file: /tmp/ws/test_results/launch_testing_ros/pytest.xml ----
======================= 8 passed, 13 warnings in 18.95s ========================
terminate called without an active exception
--- stderr: launch_testing_ros

And today it happened again: 268

test/examples/wait_for_topic_launch_test.py::wait_for_topic_launch_test
  Warning: Exception in thread Thread-8 (spin)

  Traceback (most recent call last):
    File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
      self.run()
    File "/usr/lib/python3.10/threading.py", line 953, in run
      self._target(*self._args, **self._kwargs)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 280, in spin
      self.spin_once()
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 717, in spin_once
      handler, entity, node = self.wait_for_ready_callbacks(timeout_sec=timeout_sec)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 703, in wait_for_ready_callbacks
      return next(self._cb_iter)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 604, in _wait_for_ready_callbacks
      raise ExternalShutdownException()
  rclpy.executors.ExternalShutdownException

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---- generated xml file: /tmp/ws/test_results/launch_testing_ros/pytest.xml ----
======================= 8 passed, 13 warnings in 19.37s ========================
pure virtual method called
terminate called without an active exception

[Crola1702]

Happened today in Linux Repeated: 2867

test/examples/wait_for_topic_launch_test.py::wait_for_topic_launch_test
  Warning: Exception in thread Thread-14 (spin)

  Traceback (most recent call last):
    File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
      self.run()
    File "/usr/lib/python3.10/threading.py", line 953, in run
      self._target(*self._args, **self._kwargs)
    File "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 280, in spin
      self.spin_once()
    File "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 717, in spin_once
      handler, entity, node = self.wait_for_ready_callbacks(timeout_sec=timeout_sec)
    File "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 703, in wait_for_ready_callbacks
      return next(self._cb_iter)
    File "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 604, in _wait_for_ready_callbacks
      raise ExternalShutdownException()
  rclpy.executors.ExternalShutdownException

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
terminate called without an active exception
---
Failed   <<< launch_testing_ros [23.2s, exited with code -6]

[Crola1702]

Happened today in aarch Release: 2184

test/examples/wait_for_topic_launch_test.py: 2 warnings
  Warning: There is no current event loop

test/examples/wait_for_topic_launch_test.py::wait_for_topic_launch_test
  Warning: Exception in thread Thread-8 (spin)

  Traceback (most recent call last):
    File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
      self.run()
    File "/usr/lib/python3.10/threading.py", line 953, in run
      self._target(*self._args, **self._kwargs)
    File "/home/jenkins-agent/workspace/nightly_linux-aarch64_release/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 280, in spin
      self.spin_once()
    File "/home/jenkins-agent/workspace/nightly_linux-aarch64_release/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 717, in spin_once
      handler, entity, node = self.wait_for_ready_callbacks(timeout_sec=timeout_sec)
    File "/home/jenkins-agent/workspace/nightly_linux-aarch64_release/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 703, in wait_for_ready_callbacks
      return next(self._cb_iter)
    File "/home/jenkins-agent/workspace/nightly_linux-aarch64_release/ws/install/rclpy/lib/python3.10/site-packages/rclpy/executors.py", line 604, in _wait_for_ready_callbacks
      raise ExternalShutdownException()
  rclpy.executors.ExternalShutdownException

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
pure virtual method called
terminate called without an active exception

[Crola1702]

Happened today in aarch64 debug: 2229

[Crola1702]

Today it happened for the first time in Rolling Nightly Release: 321

Log output:

--- stderr: launch_testing_ros

=============================== warnings summary ===============================
test/test_flake8.py::test_flake8
test/test_flake8.py::test_flake8
  Warning: SelectableGroups dict interface is deprecated. Use select.

test/examples/check_msgs_launch_test.py: 2 warnings
test/examples/check_node_launch_test.py: 2 warnings
test/examples/set_param_launch_test.py: 2 warnings
test/examples/talker_listener_launch_test.py: 2 warnings
test/examples/wait_for_topic_launch_test.py: 2 warnings
  Warning: There is no current event loop

test/examples/wait_for_topic_launch_test.py::wait_for_topic_launch_test
  Warning: Exception in thread Thread-8 (spin)

  Traceback (most recent call last):
    File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
      self.run()
    File "/usr/lib/python3.10/threading.py", line 953, in run
      self._target(*self._args, **self._kwargs)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 280, in spin
      self.spin_once()
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 717, in spin_once
      handler, entity, node = self.wait_for_ready_callbacks(timeout_sec=timeout_sec)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 703, in wait_for_ready_callbacks
      return next(self._cb_iter)
    File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 604, in _wait_for_ready_callbacks
      raise ExternalShutdownException()
  rclpy.executors.ExternalShutdownException

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
terminate called without an active exception
---
Failed   <<< launch_testing_ros [20.2s, exited with code -6]

[Crola1702]

It's happening consistently in CycloneDDS since the last week.

launch_testing_ros test phase fails because terminate called without an active exception.

Reference build: https://build.ros2.org/view/Rci/job/Rci__nightly-cyclonedds_ubuntu_jammy_amd64/386/

Log output:

``` Starting >>> launch_testing_ros ============================= test session starts ============================== platform linux -- Python 3.10.6, pytest-7.2.2, pluggy-1.0.0 cachedir: /tmp/ws/build_isolated/launch_testing_ros/.pytest_cache rootdir: /tmp/ws/src/ros2/launch_ros/launch_testing_ros, configfile: pytest.ini, testpaths: test plugins: launch-testing-ros-0.23.0, launch-testing-1.4.1, ament-copyright-0.14.0, ament-pep257-0.14.0, ament-flake8-0.14.0, ament-lint-0.14.0, rerunfailures-11.1.2, timeout-2.1.0, mock-3.6.1, colcon-core-0.12.1, cov-3.0.0 collected 8 items test/test_copyright.py . [ 12%] test/test_flake8.py . [ 25%] test/test_pep257.py . [ 37%] test/examples/check_msgs_launch_test.py . [ 50%] test/examples/check_node_launch_test.py . [ 62%] test/examples/set_param_launch_test.py . [ 75%] test/examples/talker_listener_launch_test.py . [ 87%] test/examples/wait_for_topic_launch_test.py . [100%] =============================== warnings summary =============================== test/test_flake8.py::test_flake8 test/test_flake8.py::test_flake8 Warning: SelectableGroups dict interface is deprecated. Use select. test/examples/check_msgs_launch_test.py: 2 warnings test/examples/check_node_launch_test.py: 2 warnings test/examples/set_param_launch_test.py: 2 warnings test/examples/talker_listener_launch_test.py: 2 warnings test/examples/wait_for_topic_launch_test.py: 2 warnings Warning: There is no current event loop test/examples/wait_for_topic_launch_test.py::wait_for_topic_launch_test Warning: Exception in thread Thread-8 (spin) Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 280, in spin self.spin_once() File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 717, in spin_once handler, entity, node = self.wait_for_ready_callbacks(timeout_sec=timeout_sec) File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 703, in wait_for_ready_callbacks return next(self._cb_iter) File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 604, in _wait_for_ready_callbacks raise ExternalShutdownException() rclpy.executors.ExternalShutdownException -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ---- generated xml file: /tmp/ws/test_results/launch_testing_ros/pytest.xml ---- ======================= 8 passed, 13 warnings in 21.50s ======================== terminate called without an active exception --- stderr: launch_testing_ros =============================== warnings summary =============================== test/test_flake8.py::test_flake8 test/test_flake8.py::test_flake8 Warning: SelectableGroups dict interface is deprecated. Use select. test/examples/check_msgs_launch_test.py: 2 warnings test/examples/check_node_launch_test.py: 2 warnings test/examples/set_param_launch_test.py: 2 warnings test/examples/talker_listener_launch_test.py: 2 warnings test/examples/wait_for_topic_launch_test.py: 2 warnings Warning: There is no current event loop test/examples/wait_for_topic_launch_test.py::wait_for_topic_launch_test Warning: Exception in thread Thread-8 (spin) Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 280, in spin self.spin_once() File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 717, in spin_once handler, entity, node = self.wait_for_ready_callbacks(timeout_sec=timeout_sec) File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 703, in wait_for_ready_callbacks return next(self._cb_iter) File "/tmp/ws/install_isolated/rclpy/local/lib/python3.10/dist-packages/rclpy/executors.py", line 604, in _wait_for_ready_callbacks raise ExternalShutdownException() rclpy.executors.ExternalShutdownException -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html terminate called without an active exception --- Failed <<< launch_testing_ros [22.3s, exited with code -6] ```

[Blast545]

@Crola1702 Is this still happening?

[fujitatomoya]

@Blast545 @Crola1702 can we close this? this is pretty old and looks like we have not seen this issue for a long time?

[Crola1702]

can we close this? this is pretty old and looks like we have not seen this issue for a long time?

Yes, I haven't seen it for a while. Closing now