new test failures for osx release nightlies and windows release nightlies

ros2 / build_farmer

0 stars 0 forks source link

new test failures for osx release nightlies and windows release nightlies #29

Closed wjwwood closed 7 years ago

wjwwood commented 7 years ago

The tests that are failing are all test communication tests between rclpy and rclcpp, some facts about them:

affects only macOS and Windows release and repeated nightlies
- Windows Debug is still failing to buld
- macOS debug is green
affects both fastrtps and connext
started June 9th, June 8th is clean
- search for merged prs: https://github.com/search?p=1&q=user%3Aament+user%3Aros2+merged%3A%222017-06-08T22%3A00%3A00-08%3A00+..+2017-06-12T23%3A30%3A00-08%3A00%22&type=Issues&utf8=✓
fails due to timeout (15 sec for most), so probably hanging

Things I have tried:

macOS CI passes
running the ci scripts on my macbook with same settings as macOS release nightly passes too

Right now I'm trying to get a new run of the job to fail, then I'll take that machine off line while the workspace is in place and rerun on that machine. Other ideas are welcome.

dhood commented 7 years ago

I also noticed that the one on mini2 had 22 fewer test failures than the ones on mini1 (e.g. http://ci.ros2.org/view/nightly/job/nightly_osx_release/487/), so it might depend on machine configs/performance somehow

wjwwood commented 7 years ago

I'm currently running on mini2, to try and get a failing workspace in place.

The delta is currently at +140 failing tests, so maybe the 22 fewer tests is a red herring. I'll look into it though.

clalancette commented 7 years ago

By the way, of the merged (currently 17) PRs, 11 of them can't possibly be related because they are only built for the turtlebot demo (and thus, only on Linux). That includes anything in the ros_astra_camera repository, the ros2/ci change, the turtlebot2_demo change, the cartographer_ros change, and the joystick_drivers change. Thus, it should be easier to focus on the remaining 6.

wjwwood commented 7 years ago

Thanks @clalancette, I was already narrowing down the pr's in this way, but it's a good point.

At this point I'm thinking that none of the pr's are causing this. I think it must be a machine configuration or something else. I'll continue to look into it.

wjwwood commented 7 years ago

I think I found the issue, which is related to uninitialized memory in rclpy since adding the new QoS setting that avoids ros name conventions.

I wasn't considering it originally because I think I had the dates slightly too strict in the search parameters. I'm trying out a fix now.