Some packages fail when running colcon test in the local environment

Checklist

[X] I've read the contribution guidelines.
[X] I've searched other issues and no duplicate issues were found.
[X] I'm convinced that this is not my fault but a bug.

Description

We are trying to visualize the test coverage of each package in autoware.universe with @yhisaki . Specifically, the current approach is to first run tests locally on packages under the planning directory, and then use codecov to visualize the test results. When running tests locally, we've confirmed that tests for some packages fail. To my understanding, in autoware.universe, having all colcon tests succeed is a required condition for merging PRs, so this shouldn't occur even on locally. Therefore, I think there might be an issue with how we're running the tests locally.

Possible reasons could be:

There's a problem with the test execution method
There's a valid reason for test results to fail locally

If anyone has insights on this, please let me know.

Expected behavior

pass all test

Actual behavior

When executing

colcon test --event-handlers console_cohesion+ --base-paths ./src/autoware/universe/

to run tests for packages in autoware.universe locally, some tests fail. We've also noticed that results vary between executions and when run on different PCs.

To mimic the colcon test behavior in GitHub Actions, we tried:

colcon lcov-result --zero-counters --verbose
colcon lcov-result --initial
colcon test --executor sequential --base-paths ./src/universe/autoware.universe/

referencing https://github.com/autowarefoundation/autoware-github-actions/blob/main/colcon-test/action.yaml However, one package (autoware_livox_tag_filter) still fails.

Steps to reproduce

Execute the following commands to run colcon test in the local environment:

colcon lcov-result --zero-counters --verbose
colcon lcov-result --initial
colcon test --executor sequential --base-paths ./src/universe/autoware.universe/

Note: You may need to install lcov via apt if you haven't already when using this method.

Versions

No response

Possible causes

No response

Additional context

No response

autowarefoundation / autoware.universe