:farmer: TestServiceIntrospection.service_introspection_enable_disable_events failing in Rolling, Jazzy and Iron Connext

Crola1702 commented 3 months ago

Bug report

Required Info:

Operating System:
- Ubuntu Noble
Installation type:
- Source
Version or commit hash:
- Rolling
DDS implementation:
- RTI Connext
Client library (if applicable):
- Rclcpp

Steps to reproduce issue

Run a build in Rci__nightly-connext_ubuntu_noble_amd64
See test regression fail

Additional information

Reference build:

Log output:

[ RUN      ] TestServiceIntrospection.service_introspection_enable_disable_events
RTI Connext DDS Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User license@rti.com For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
/tmp/ws/src/ros2/rclcpp/rclcpp/test/rclcpp/test_service_introspection.cpp:201: Failure
Expected equality of these values:
  events.size()
    Which is: 1
  2U
    Which is: 2

/tmp/ws/src/ros2/rclcpp/rclcpp/test/rclcpp/test_service_introspection.cpp:218: Failure
Expected equality of these values:
  events.size()
    Which is: 3
  4U
    Which is: 4

[  FAILED  ] TestServiceIntrospection.service_introspection_enable_disable_events (3309 ms)
[ RUN      ] TestServiceIntrospection.service_introspection_enable_disable_event_content
RTI Connext DDS Non-commercial license is for academic, research, evaluation and personal use only. USE FOR COMMERCIAL PURPOSES IS PROHIBITED. See RTI_LICENSE.TXT for terms. Download free tools at rti.com/ncl. License issued to Non-Commercial User license@rti.com For non-production use only.
Expires on 00-jan-00 See www.rti.com for more information.
/tmp/ws/src/ros2/rclcpp/rclcpp/test/rclcpp/test_service_introspection.cpp:340: Failure
Expected equality of these values:
  events.size()
    Which is: 3
  4U
    Which is: 4

[  FAILED  ] TestServiceIntrospection.service_introspection_enable_disable_event_content (1307 ms)

It's a flaky with a high percentage failure rate: job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
Ici__nightly-connext_ubuntu_jammy_amd64	2024-08-28	2024-08-14	6	4	66.67
Rci__nightly-connext_ubuntu_noble_amd64	2024-08-27	2024-08-14	12	11	91.67
Jci__nightly-connext_ubuntu_noble_amd64	2024-08-26	2024-08-13	6	5	83.33

fujitatomoya commented 2 months ago

i cannot reproduce this issue with my local environment.

root@tomoyafujita:~/ros2_ws/colcon_ws# RMW_IMPLEMENTATION=rmw_connextdds
root@tomoyafujita:~/ros2_ws/colcon_ws# colcon test --event-handlers console_direct+ --retest-until-fail 100 --packages-select rclcpp --ctest-args -R test_service_introspection
...
The following tests passed:
        test_service_introspection

100% tests passed, 0 tests failed out of 1

Label Time Summary:
gmock     =   1.24 sec*proc (1 test)
mimick    =   1.24 sec*proc (1 test)

Total Test time (real) = 125.27 sec
Finished <<< rclcpp [2min 6s]

Summary: 1 package finished [2min 7s]

according to the log information in the report, client events are okay as expected, but server side missing one event. and we can see at least one service event is received from service server, that means publisher and topic are created okay. it would be worth to try to extend the timeout from 1000msec to 3000msec, but i am not sure if that actually fixes the problem.

root cause of this issue seems like almost same with https://github.com/ros2/rclcpp/issues/2588#issuecomment-2259355642, that is dropping the messages (some messages are missing, but some are okay) using same publisher object.

fujitatomoya commented 1 month ago

@fgallegosalido this test also fails with rmw_connextdds only, can you take a look at my comment on https://github.com/ros2/rclcpp/pull/2651? i was trying to address this unstable problem, but it does not work as expected.

fgallegosalido commented 1 month ago

I will look into it and let you know what I find.

fujitatomoya commented 1 month ago

@fgallegosalido just checking, any updates on this?

fgallegosalido commented 4 weeks ago

@fujitatomoya sorry, not much was done. I was able to reproduce the issue and started working on it, but couldn't find the root cause, so I moved to do other things. I will resume the work on it as soon as possible

fujitatomoya commented 4 weeks ago

@fgallegosalido no worries, thanks for working on this. i will revisit https://github.com/ros2/rclcpp/pull/2651 to see if i can find some clue on this.

ros2 / rclcpp