Test refactoring part 2

gazebosim / gz-transport

Transport library for component communication based on publication/subscription and service calls.

https://gazebosim.org

Apache License 2.0

29 stars 36 forks source link

Test refactoring part 2 #463

Closed mjcarroll closed 5 months ago

mjcarroll commented 7 months ago

Further cleanups to testing after the subprocess changes.

This does:

Pushes auxillary test executables down one directory, just to tidy up the integration folder and makes it clear which things are tests vs support executables
Splits "test utils" (utility code) from "test config" (generated configuration)
Puts auxillary executable definitions in one location

codecov[bot] commented 7 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (ca954cb) 87.23% compared to head (b9fb754) 87.27%. Report is 1 commits behind head on gz-transport12.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## gz-transport12 #463 +/- ## ================================================== + Coverage 87.23% 87.27% +0.03% ================================================== Files 60 60 Lines 5211 5211 ================================================== + Hits 4546 4548 +2 + Misses 665 663 -2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

mjcarroll commented 7 months ago

There is still a failing homebrew test (that I think is new) and also Windows is failing to clone the repo (@j-rivero?)

mjcarroll commented 7 months ago

@osrf-jenkins retest this please

mjcarroll commented 6 months ago

I found the culprit here, I accidentally switched out the test executables from one that runs indefinitely to one that exits after a few seconds. It seems that on Linux runners, we are fast enough to complete the stress test in the window, where on other platforms we are too slow.

I have switched the test back accordingly.

mjcarroll commented 6 months ago

@osrf-jenkins retest this please

caguero commented 6 months ago

Do you know if the Windows failure is related to this PR?

mjcarroll commented 6 months ago

Do you know if the Windows failure is related to this PR?

I'm not sure. On one hand I haven't seen it before, so maybe my fault, but also seems unrelated to the changes in the PR?

mjcarroll commented 6 months ago

Looks like there are actual multiple failing tests, but they aren't being reported correctly:

The following tests FAILED:
     47 - INTEGRATION_twoProcsSrvCallStress (Timeout)
     53 - INTEGRATION_twoProcsSrvCallWithoutInputStress (Timeout)
     59 - INTEGRATION_twoProcsSrvCallWithoutOutputStress (Timeout)
     83 - INTEGRATION_playback (Failed)
Errors while running CTest

mjcarroll commented 5 months ago

Looks like Windows failure may have been a flake. I will merge this and continue to monitor.