search

gazebosim / gz-transport

Transport library for component communication based on publication/subscription and service calls.
https://gazebosim.org
Apache License 2.0
29 stars 37 forks source link

👩‍🌾 ReplayStep test regression in Windows #195

Open chapulina opened 3 years ago

chapulina commented 3 years ago

This test used to pass consistently, but started failing since 2020-10-23 for ign-transport8:

https://build.osrfoundation.org/job/ignition_transport-ci-ign-transport8-windows7-amd64/28/testReport/(root)/playback/ReplayStep/history/

It passes on ign-transport9 though:

https://build.osrfoundation.org/job/ignition_transport-ci-ign-transport9-windows7-amd64/1/testReport/(root)/playback/ReplayStep/history/

Failure log:

79: [ RUN      ] playback.ReplayStep
79: Chirping [100] times on [3] topics:
79:  -- /foo
79:  -- /bar
79:  -- /baz
79: Stepping playback...
79: D:\Jenkins\workspace\ignition_transport-ci-ign-transport8-windows7-amd64\ws\ign-transport\log\test\integration\playback.cc(660): error: Value of: MessagesAreEqual(secondMessageData, thirdMessageData)
79:   Actual: true
79: Expected: false
79: Waiting to for playback to finish...
79:  Done waiting...
79: Playback finished!
79: [  FAILED  ] playback.ReplayStep (4688 ms)
Crola1702 commented 1 year ago

I entered the test regression in buildfarmer database and found out that this issue happened less than 25 times. It has happened in the following jobs:

Job Last time seen First time seen (Since 2021)
ign_transport-ign-8-win 2023-01-24 2021-05-01
ign_transport-gz-12-win 2022-12-01 2022-08-31
ign_transport-ign-11-win 2022-09-14 2022-05-25
ign_transport-ci-win 2022-08-02 2022-08-02

This means that this issue is happening on Citadel, Fortress and Garden

Crola1702 commented 10 months ago

This regression is still happening flaky on transport-11 and transport-13

Flaky ratio in the last 30 days:

job_name last_fail first_fail build_count failure_count failure_percentage
ign_transport-ign-11-win 2023-10-20 2023-10-20 7 1 14.29
ign_transport-gz-13-win 2023-10-17 2023-10-08 10 2 20.0

FYI @azeey

Crola1702 commented 8 months ago

Flaky ration in the last 30 days:

job_name last_fail first_fail build_count failure_count failure_percentage
gz_transport-ign-transport11-win 2024-01-07 2023-12-18 7 2 28.57
gz_transport-ign-transport8-win 2024-01-03 2024-01-03 7 1 14.29
gz_transport-12-win 2023-12-30 2023-12-17 7 3 42.86
gz_transport-13-win 2023-12-26 2023-12-12 8 2 25.0
Crola1702 commented 1 month ago

This issue seems to be happening more often this week.

Flaky ratio in the last 15 days:

job_name last_fail first_fail build_count failure_count failure_percentage
gz_transport-main-win 2024-08-03 2024-07-30 4 2 50.0
gz_transport-ign-transport8-win 2024-08-03 2024-08-03 4 1 25.0
gz_transport-12-win 2024-07-25 2024-07-25 4 1 25.0
Crola1702 commented 1 month ago

Log output is almost the same as before:

83: [ RUN      ] playback.ReplayStep
83: Stepping playback...
83: C:\J\workspace\gz_transport-main-win\ws\gz-transport\log\test\integration\playback.cc(652): error: Value of: MessagesAreEqual(firstMessageData, secondMessageData)
83:   Actual: true
83: Expected: false
83: Waiting to for playback to finish...
83:  Done waiting...
83: Playback finished!
83: [  FAILED  ] playback.ReplayStep (11265 ms)