search

gazebosim / gz-sim

Open source robotics simulator. The latest version of Gazebo.
https://gazebosim.org
Apache License 2.0
732 stars 273 forks source link

Flaky / failing / disabled tests #8

Open osrf-migration opened 5 years ago

osrf-migration commented 5 years ago

Original report (archived issue) by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

This issue collects all known flaky tests, we should update it as we observe new flaky tests or if they stop flaking.

:snowflake: :snowflake: :snowflake:

PERFORMANCE_each

Platforms: Jenkins Ubuntu and Pipelines

Failures:

39: [ RUN      ] EntityComponentManagerPerfrormance.Each
39: /opt/atlassian/pipelines/agent/build/test/performance/each.cc:141: Failure
39: Expected: (cacheEntityAvg) < (cachelessEntityAvg), actual: 7200.83 vs 7170.62
39: [  FAILED  ] EntityComponentManagerPerfrormance.Each (60561 ms)

UNIT_SimulationRunner_TEST

Platforms: Pipelines

Failures:

19: /opt/atlassian/pipelines/agent/build/src/SimulationRunner_TEST.cc:958: Failure
19: Value of: simTime.second
19:   Actual: 100000000
19: Expected: clockMsgs.back().mutable_sim()->nsec()
19: Which is: 99000000

INTEGRATION_network_handshake

Platforms: Homebrew

Failures:

/Users/jenkins/workspace/ignition_gazebo-ci-pr_any-homebrew-amd64/ign-gazebo/test/integration/network_handshake.cc:56
Value of: success
  Actual: 4-byte object <01-00 00-00>
Expected: std::cv_status::no_timeout
Which is: 4-byte object <00-00 00-00>

Platforms: Pipelines

Failures:

59: [ RUN      ] NetworkHandshake.Handshake
59: [Err] [LevelManager.cc:108] Could not find a plugin tag with name ignition::gazebo. Levels and distributed simulation will not work.
59: [Err] [LevelManager.cc:108] Could not find a plugin tag with name ignition::gazebo. Levels and distributed simulation will not work.
59: [Err] [LevelManager.cc:108] Could not find a plugin tag with name ignition::gazebo. Levels and distributed simulation will not work.
59/80 Test #59: INTEGRATION_network_handshake ................***Timeout 240.01 sec

Platforms: Jenkins Ubuntu

Date added: 2019-04-11

Failures:

61: [----------] 2 tests from NetworkHandshake
61: [ RUN      ] NetworkHandshake.Handshake
61: [Msg] Loading SDF string.
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2251689575469537287] / [ign_gazebo_components.World].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13994732549916512682] / [ign_gazebo_components.Name].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [12592746352568925681] / [ign_gazebo_components.Gravity].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13224937992534617849] / [ign_gazebo_components.MagneticField].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [8064491505919932473] / [ign_gazebo_components.Level].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2668898242563798256] / [ign_gazebo_components.DefaultLevel].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [11371360182141354106] / [ign_gazebo_components.LevelEntityNames].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [3297509811873971798] / [ign_gazebo_components.ParentEntity].
61: [Msg] Serving scene information on [/world/default/scene/info]
61: [Msg] Serving graph information on [/world/default/scene/graph]
61: [Msg] Serving scene information on [/world/default/scene/info]
61: [Msg] Publishing entity deletions on [/world/default/scene/deletion]
61: [Msg] Publishing pose messages on [/world/default/pose/info]
61: [Dbg] [SimulationRunner.cc:543] Loaded system [ignition::gazebo::systems::SceneBroadcaster] for entity [1]
61: [Msg] Create service on [/world/default/create]
61: [Msg] Remove service on [/world/default/remove]
61: [Dbg] [SimulationRunner.cc:543] Loaded system [ignition::gazebo::systems::UserCommands] for entity [1]
61: [Msg] Network Primary, expects [2] secondaries.
61: [Msg] Serving GUI information on [/world/default/gui/info]
61: [Msg] World [default] initialized with [default_physics] physics profile.
61: [Msg] Loading SDF string.
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2251689575469537287] / [ign_gazebo_components.World].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13994732549916512682] / [ign_gazebo_components.Name].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [12592746352568925681] / [ign_gazebo_components.Gravity].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13224937992534617849] / [ign_gazebo_components.MagneticField].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [8064491505919932473] / [ign_gazebo_components.Level].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2668898242563798256] / [ign_gazebo_components.DefaultLevel].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [11371360182141354106] / [ign_gazebo_components.LevelEntityNames].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [3297509811873971798] / [ign_gazebo_components.ParentEntity].
61: [Msg] Serving scene information on [/world/default/scene/info]
61: [Msg] Serving graph information on [/world/default/scene/graph]
61: [Msg] Serving scene information on [/world/default/scene/info]
61: [Msg] Publishing entity deletions on [/world/default/scene/deletion]
61: [Msg] Publishing pose messages on [/world/default/pose/info]
61: [Dbg] [SimulationRunner.cc:543] Loaded system [ignition::gazebo::systems::SceneBroadcaster] for entity [1]
61: [Msg] Create service on [/world/default/create]
61: [Msg] Remove service on [/world/default/remove]
61: [Dbg] [SimulationRunner.cc:543] Loaded system [ignition::gazebo::systems::UserCommands] for entity [1]
61: [Dbg] [NetworkManagerSecondary.cc:49] Advertised PeerControl service on [d235de65/control]
61: [Msg] Network Secondary, with namespace [d235de65].
61: [Msg] Serving GUI information on [d235de65/world/default/gui/info]
61: [Msg] World [default] initialized with [default_physics] physics profile.
61: [Msg] Loading SDF string.
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2251689575469537287] / [ign_gazebo_components.World].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13994732549916512682] / [ign_gazebo_components.Name].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [12592746352568925681] / [ign_gazebo_components.Gravity].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [13224937992534617849] / [ign_gazebo_components.MagneticField].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [8064491505919932473] / [ign_gazebo_components.Level].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [2668898242563798256] / [ign_gazebo_components.DefaultLevel].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [11371360182141354106] / [ign_gazebo_components.LevelEntityNames].
61: [Dbg] [EntityComponentManager.cc:544] Using components of type [3297509811873971798] / [ign_gazebo_components.ParentEntity].
61: [Msg] Serving scene information on [/world/default/scene/info[[WWrn] [Componet[WWrn] [ComponetTrying to serialize copmponent of type [Trying to serialize copmponent of type [22516895754695372872251689575469537287], which hasn't implemented the `Serialize` function. Component will not be serialized.
61: ], which hasn't implemented the `Serialize` function. Component will not be serialized.
61/82 Test #61: INTEGRATION_network_handshake ................***Exception: SegFault  3.07 sec

INTEGRATION_physics_system

Platforms: Homebrew

Failures

Unable to find test results for INTEGRATION_physics_system.xml, test did not run. Expected results in /Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo3-homebrew-amd64/build/test_results/INTEGRATION_physics_system.xml

Date added: 2020-07-10

UNIT_PeerTracker_TEST

Platforms: Homebrew

Failures:

/Users/jenkins/workspace/ignition_gazebo-ci-pr_any-homebrew-amd64/ign-gazebo/src/network/PeerTracker_TEST.cc:258
Value of: tracker3.NumPeers()
  Actual: 0
Expected: 1u
Which is: 1
/Users/jenkins/workspace/ignition_gazebo-ci-pr_any-homebrew-amd64/ign-gazebo/src/network/PeerTracker_TEST.cc:84
Value of: tracker5->NumPeers()
  Actual: 2
Expected: 5u
Which is: 5
/Users/jenkins/workspace/ignition_gazebo-ci-pr_any-homebrew-amd64/ign-gazebo/src/network/PeerTracker_TEST.cc:163
Value of: stalePeers
  Actual: 0
Expected: 1
/Users/jenkins/workspace/ignition_gazebo-ci-ign-gazebo2-homebrew-amd64/ign-gazebo/src/network/PeerTracker_TEST.cc:167
Value of: tracker1->NumPeers()
  Actual: 1
Expected: 0u
Which is: 0

UNIT_NetworkManager_TEST

Platforms: Pipelines

Failures:

31: [ RUN      ] NetworkManager.Step
31: [Wrn] [NetworkManagerPrimary.cc:69] NetworkManager started without EventManager. Distributed environment may not terminate correctly
31/82 Test #31: UNIT_NetworkManager_TEST .....................***Exception: SegFault  1.15 sec
osrf-migration commented 5 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

osrf-migration commented 5 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

osrf-migration commented 5 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

osrf-migration commented 5 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

osrf-migration commented 5 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

osrf-migration commented 5 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

osrf-migration commented 5 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

INTEGRATION_gpu_lidar is not flaky on homebrew; it is failing consistently. I'll move it to its own issue

osrf-migration commented 5 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

sometimes INTEGRATION_gpu_lidar doesn't get run because the build node doesn't have an X server running, which makes it seem like a flaky test, but it's really not

osrf-migration commented 5 years ago

Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

chapulina commented 4 years ago

UNIT_Gui_TEST

Platforms: Homebrew

Failure: segfault at the very start

39: [----------] 1 test from GuiTest
39: [ RUN      ] GuiTest.PathManager
 39/128 Test  #39: UNIT_Gui_TEST ........................................***Exception: SegFault  0.05 sec
test 40
        Start  40: check_UNIT_Gui_TEST
azeey commented 4 years ago

INTEGRATION_pose_publisher_system

Platforms: ign-gazebo3 Homebrew

Failure: Segfault on PosePublisherTest.StaticPosePublisher.

103: [ RUN      ] PosePublisherTest.StaticPosePublisher
103/134 Test #103: INTEGRATION_pose_publisher_system ....................***Exception: SegFault  5.94 sec
test 104
        Start 104: check_INTEGRATION_pose_publisher_system

INTEGRATION_multicopter

Platforms: ign-gazebo2 Homebrew

Failure: Segfault on MulticopterTest.MulticopterVelocityControl

89: [       OK ] MulticopterTest.CommandedMotorSpeed (1090 ms)
89: [ RUN      ] MulticopterTest.MulticopterVelocityControl
 89/130 Test  #89: INTEGRATION_multicopter ..............................***Exception: SegFault  2.00 sec
test 90
        Start  90: check_INTEGRATION_multicopter
tfoote commented 3 years ago

ign-gazebo5 Homebrew https://build.osrfoundation.org/job/ignition_gazebo-ci-ign-gazebo5-homebrew-amd64/29/consoleFull

The following tests FAILED:
    103 - INTEGRATION_lift_drag_system (SEGFAULT)
    153 - INTEGRATION_user_commands (SEGFAULT)
Errors while running CTest

Also further documented at https://github.com/osrf/buildfarmer/issues/168 closing that in favor of this

INTEGRATION_wheel_slip
INTEGRATION_network_handshake
INTEGRATION_thermal_system