search

gazebosim / gz-rendering

C++ library designed to provide an abstraction for different rendering engines. It offers unified APIs for creating 3D graphics applications.
https://gazebosim.org
Apache License 2.0
50 stars 48 forks source link

:farmer: `SEH Exceptions` in gz-rendering-win #745

Open Crola1702 opened 1 year ago

Crola1702 commented 1 year ago

Environment

Description

Steps to reproduce

  1. Run a build on gz-rendering7-win, gz-rendering8-win or ign_rendering-ci-win
  2. See it fail

Output

We have 65 test regressions in gz-rendering windows (7 and main)

First time we saw it: Sept 1 It stopped between Sept 28 and Oct 11

And now it came back

We think it's probably related to this issue: https://stackoverflow.com/questions/13157671/seh-exception-with-code-0xc0000005-thrown-in-the-test-body

Update (17/08/2023):

Reference build: https://build.osrfoundation.org/job/ign_rendering-ci-win/254/#showFailuresLink

Test regressions * [CameraTest.ViewProjectionMatrix](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/CameraTest/ViewProjectionMatrix/) * [CameraTest.RenderTexture](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/CameraTest/RenderTexture/) * [CameraTest.TrackFollow](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/CameraTest/TrackFollow/) * [CameraTest.AddRemoveRenderPass](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/CameraTest/AddRemoveRenderPass/) * [CameraTest.VisibilityMask](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/CameraTest/VisibilityMask/) * [CameraTest.IntrinsicMatrix](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/CameraTest/IntrinsicMatrix/) * [GpuRaysTest.Configure](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/GpuRaysTest/Configure/) * [LidarVisualTest.Configure](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/LidarVisualTest/Configure/) * [LidarVisualTest.LidarVisual](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/LidarVisualTest/LidarVisual/) * [ProjectorTest.Projector](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/ProjectorTest/Projector/) * [SceneTest.Scene](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/SceneTest/Scene/) * [SceneTest.SceneGradient](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/SceneTest/SceneGradient/) * [SceneTest.Nodes](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/SceneTest/Nodes/) * [SceneTest.RemoveNodes](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/SceneTest/RemoveNodes/) * [SceneTest.DestroyNodes](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/SceneTest/DestroyNodes/) * [SceneTest.NodeCycle](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/SceneTest/NodeCycle/) * [SceneTest.Materials](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/SceneTest/Materials/) * [SceneTest.Time](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/SceneTest/Time/) * [SceneTest.BackgroundMaterial](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/SceneTest/BackgroundMaterial/) * [ThermalCameraTest.ThermalCamera](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/ThermalCameraTest/ThermalCamera/) * [ReloadEngineTest.Scene](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/ReloadEngineTest/Scene/) * [ArrowVisualTest.ArrowVisual](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/ArrowVisualTest/ArrowVisual/) * [AxisVisualTest.AxisVisual](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/AxisVisualTest/AxisVisual/) * [COMVisualTest.COMVisual](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/COMVisualTest/COMVisual/) * [CapsuleTest.Capsule](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/CapsuleTest/Capsule/) * [GizmoVisualTest.GizmoVisual](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/GizmoVisualTest/GizmoVisual/) * [GizmoVisualTest.Material](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/GizmoVisualTest/Material/) * [GridTest.Grid](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/GridTest/Grid/) * [InertiaVisualTest.InertiaVisual](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/InertiaVisualTest/InertiaVisual/) * [LightVisualTest.LightVisual](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/LightVisualTest/LightVisual/) * [LightTest.Light](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/LightTest/Light/) * [MarkerTest.Marker](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/MarkerTest/Marker/) * [MaterialTest.MaterialProperties](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/MaterialTest/MaterialProperties/) * [MeshDescriptorTest.Descriptor](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/MeshDescriptorTest/Descriptor/) * [MeshTest.MeshSubMesh](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/MeshTest/MeshSubMesh/) * [MeshTest.MeshSkeleton](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/MeshTest/MeshSkeleton/) * [MeshTest.MeshSkeletonAnimation](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/MeshTest/MeshSkeletonAnimation/) * [MeshTest.MeshClone](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/MeshTest/MeshClone/) * [MoveToHelperTest.MoveTo](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/MoveToHelperTest/MoveTo/) * [NodeTest.Pose](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/NodeTest/Pose/) * [OrbitViewControllerTest.OrbitViewControl](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/OrbitViewControllerTest/OrbitViewControl/) * [OrbitViewControllerTest.Constructor](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/OrbitViewControllerTest/Constructor/) * [OrbitViewControllerTest.Control](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/OrbitViewControllerTest/Control/) * [OrthoViewControllerTest.OrthoViewControl](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/OrthoViewControllerTest/OrthoViewControl/) * [OrthoViewControllerTest.Control](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/OrthoViewControllerTest/Control/) * [ParticleEmitterTest.ParticleEmitter](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/ParticleEmitterTest/ParticleEmitter/) * [RayQueryTest.RayQuery](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/RayQueryTest/RayQuery/) * [RenderTargetTest.RenderTexture](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/RenderTargetTest/RenderTexture/) * [RenderTargetTest.RenderWindow](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/RenderTargetTest/RenderWindow/) * [RenderTargetTest.AddRemoveRenderPass](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/RenderTargetTest/AddRemoveRenderPass/) * [RenderingIfaceTest.GetEngine](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/RenderingIfaceTest/GetEngine/) * [TextTest.Text](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/TextTest/Text/) * [TransformControllerTest.TransformControl](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/TransformControllerTest/TransformControl/) * [TransformControllerTest.WorldSpace](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/TransformControllerTest/WorldSpace/) * [TransformControllerTest.LocalSpace](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/TransformControllerTest/LocalSpace/) * [TransformControllerTest.Control2d](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/TransformControllerTest/Control2d/) * [VisualTest.Material](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/VisualTest/Material/) * [VisualTest.Children](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/VisualTest/Children/) * [VisualTest.Scale](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/VisualTest/Scale/) * [VisualTest.UserData](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/VisualTest/UserData/) * [VisualTest.Geometry](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/VisualTest/Geometry/) * [VisualTest.VisibilityFlags](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/VisualTest/VisibilityFlags/) * [VisualTest.BoundingBox](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/VisualTest/BoundingBox/) * [VisualTest.Wireframe](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/VisualTest/Wireframe/) * [VisualTest.Clone](https://build.osrfoundation.org/job/ign_rendering-ci-win/254/testReport/junit/(root)/VisualTest/Clone/)

Log Output:

17: [Err] [C:\J\workspace\ign_rendering-ci-win\ws\gz-rendering\ogre\src\OgreRenderEngine.cc:695] Unable to create the rendering window. Attempt 9. Exception Ogre::RenderingAPIException::RenderingAPIException: OpenGL 1.5 is not supported in GLRenderSystem::initialiseContext at C:\vcpkg\buildtrees\ogre\src\eddf310f0b-6ab1152694.clean\RenderSystems\GL\src\OgreGLRenderSystem.cpp (line 1167)
17: [Err] [C:\J\workspace\ign_rendering-ci-win\ws\gz-rendering\ogre\src\OgreRenderEngine.cc:704] Unable to create the rendering window after 10 attempts.
17: [Err] [C:\J\workspace\ign_rendering-ci-win\ws\gz-rendering\ogre\src\OgreRenderEngine.cc:638] Failed to create dummy render window.
17: [Err] [C:\J\workspace\ign_rendering-ci-win\ws\gz-rendering\ogre\src\OgreRenderEngine.cc:737] Failed to get capabilities
17: [Wrn] [C:\J\workspace\ign_rendering-ci-win\ws\gz-rendering\ogre\src\OgreRenderEngine.cc:797] Cannot initialize render engine since render path type is NONE. Ignore this warning if rendering has been turned off on purpose.

unknown file
SEH exception with code 0xc0000005 thrown in the test body.

This issue started happening again on Aug 11 on different machines

job_name last_fail first_fail build_count failure_count failure_percentage
ign_rendering-gz-7-win 2023-08-15 2023-08-11 10 3 30.0
ign_rendering-ci-win 2023-08-15 2023-08-12 9 2 22.22

Edit 17-10-2023:

New test regression (new test added by #908):

This is now a consistent failure in gz-rendering 7 and 8 Windows

iche033 commented 1 year ago

is this still an issue?

Crola1702 commented 10 months ago

I updated the issue with new information

azeey commented 9 months ago

The SEH exception might be a red herring. Based on the error

11: [Err] [C:\J\workspace\ign_rendering-gz-7-win\ws\gz-rendering\ogre\src\OgreRenderEngine.cc:695] Unable to create the rendering window. Attempt [0]. Exception [Ogre::RenderingAPIException::RenderingAPIException: OpenGL 1.5 is not supported in GLRenderSystem::initialiseContext at C:\vcpkg\buildtrees\ogre\src\eddf310f0b-6ab1152694.clean\RenderSystems\GL\src\OgreGLRenderSystem.cpp (line 1167)]

my guess is that the graphics card is not set up properly on that machine. The same test doesn't print that error on a passing test on a different machine (see INTEGRATION_boundingbox_camera_ogre_gl3plus (test 11) on https://build.osrfoundation.org/job/ign_rendering-gz-7-win/79/consoleFull#console-section-12).

mjcarroll commented 9 months ago

Yeah, the error is probably likely that something in the scene creation process is an unchecked nullptr. For the most part, you can think of the SEH 0xc0000005 as the Windows equivalent to a segfault. I believe that Windows calls it an access violation, which means that a pointer is not what it should be.

Crola1702 commented 6 months ago

This is also happening on gz_rendering-main.

I added Ionic as a label

Crola1702 commented 3 months ago

This is still affecting gz-rendering windows jobs with the same output

Reference builds:

Failure Percentage:

job_name last_fail first_fail build_count failure_count failure_percentage
gz_rendering-main-win 2024-04-08 2024-03-12 8 8 100.0
gz_rendering-7-win 2024-04-07 2024-03-10 8 8 100.0
gz_rendering-8-win 2024-04-04 2024-03-09 8 8 100.0
azeey commented 3 months ago

@Crola1702 any progress on setting up ogre 2.3 on our windows machines? I believe that's a requirement for fixing these tests.

Crola1702 commented 3 months ago

@Crola1702 any progress on setting up ogre 2.3 on our windows machines

I'm not sure. I can take a look with @j-rivero's help. Any ideas on where to start?

azeey commented 3 months ago

Last time we talked about this, I remember the next action item was to check if our vcpkg port for ogre-next-23 actually works. If you can verify that locally, I think we can make the change in https://github.com/gazebo-tooling/release-tools/blob/f392d30813b5097229b22413f56a6733556d34f4/jenkins-scripts/lib/windows_env_vars.bat#L22