search

ValveSoftware / SteamVR-for-Linux

Issue tracker for the Linux port of SteamVR
930 stars 45 forks source link

vrwebhelper crashing (radv queue submission) #278

Closed dsalt closed 4 years ago

dsalt commented 4 years ago

Describe the bug vrwebhelper crashing during, or shortly after, startup.

I tweaked its startup script so that I could get a core dump from it:

#1  0x00007fd405e2a535 in __GI_abort () at abort.c:79
#2  0x00007fd35aaef7e8 in radv_QueueSubmit () from /usr/lib/x86_64-linux-gnu/libvulkan_radeon.so
#3  0x00007fd3ffae7d32 in CVRCompositorSharedTextures::EndCommandBuffer(CVRCompositorSharedTextures::CommandBuffer_t**) () from /home/steam/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so
#4  0x00007fd3ffae940d in CVRCompositorSharedTextures::UpdateTextureSet(CVRCompositorSharedTextures::Params const*, VRCompositorState_TextureSet_t*) () from /home/steam/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so
#5  0x00007fd3ffb10d18 in CVROverlayLatest::SetOverlayTexture(unsigned long, vr::Texture_t const*) () from /home/steam/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so
#6  0x0000000000417b93 in CCefClientHandler::OnPaint(scoped_refptr<CefBrowser>, cef_paint_element_type_t, std::vector<CefRect, std::allocator<CefRect> > const&, void const*, int, int) ()
#7  0x0000000000417da1 in non-virtual thunk to CCefClientHandler::OnPaint(scoped_refptr<CefBrowser>, cef_paint_element_type_t, std::vector<CefRect, std::allocator<CefRect> > const&, void const*, int, int) ()
#8  0x0000000000511d12 in (anonymous namespace)::render_handler_on_paint(_cef_render_handler_t*, _cef_browser_t*, cef_paint_element_type_t, unsigned long, _cef_rect_t const*, void const*, int, int) ()
#9  0x00007fd408d88875 in ?? ()
#10 0x0000000000000320 in ?? ()
#11 0x0000000000000258 in ?? ()
#12 0x0000000000000320 in ?? ()
#13 0x00007fd33c027010 in ?? ()
#14 0x0000000000000258 in ?? ()
#15 0x0000000000511ba0 in ?? ()
#16 0x000000002cd35aa8 in ?? ()
#17 0x00000000025f3cb0 in ?? ()
#18 0x000000000262f930 in ?? ()
#19 0x0000000000000320 in ?? ()
#20 0x0000000000000258 in ?? ()
#21 0x0000000000000000 in ?? ()

There is related text in the kernel log:

[drm:amdgpu_syncobj_lookup_and_add_to_sync] *ERROR* syncobj 17 failed to find fence @ 0 (-22)!
[drm:amdgpu_cs_ioctl] *ERROR* Failed in the dependencies handling -22!

To Reproduce Steps to reproduce the behavior:

  1. Start up SteamVR.
  2. Check the running processes, observe no running vrwebhelper processes and one zombie instancer which was spawned by vrdashboard.

Expected behavior Running as [intended.

System Information (please complete the following information):

dsalt commented 4 years ago

Still happening, with the addition now of the settings window either not appearing or being blank. Could be related to #255.

vrwebhelper_main log

vrclient_vrwebhelper_main.log

dsalt commented 4 years ago

Also happening in some games. I tested Neos VR and got a similar backtrace (Mesa 20.0.1)

#7  0x00007fc69fa98535 in __GI_abort () at abort.c:79
#8  0x00007fc4dbbdf4ca in radv_queue_submit_deferred (processing_list=0x7fc537fee570, submission=0x7fc5095335d0)
    at ../src/amd/vulkan/radv_device.c:4545
#9  radv_process_submissions (processing_list=processing_list@entry=0x7fc537fee570)
    at ../src/amd/vulkan/radv_device.c:4638
#10 0x00007fc4dbbdf97c in radv_queue_submit (queue=queue@entry=0x7fc508f91ad0, submission=submission@entry=0x7fc537fee5e0)
    at ../src/amd/vulkan/radv_device.c:4658
#11 0x00007fc4dbbe38ed in radv_QueueSubmit (_queue=0x7fc508f91ad0, submitCount=1, pSubmits=<optimized out>, fence=0x7fc5094ab950)
    at ../src/amd/vulkan/radv_device.c:4707
#12 0x00007fc5342870c2 in CVRCompositorSharedTextures::EndCommandBuffer(CVRCompositorSharedTextures::CommandBuffer_t**) ()
   from /home/Steam/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so
#13 0x00007fc53427d289 in CVRCompositorClient::Submit(vr::EVREye, vr::Texture_t const*, vr::VRTextureBounds_t const*, vr::EVRSubmitFlags) ()
   from /home/Steam/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so
#14 0x00007fc6a0d3bb58 in OpenVR::SubmitOneEyeTexture(unsigned int) ()
   from /home/Steam/Steam/steamapps/common/NeosVR/UnityPlayer.so
#15 0x00007fc6a0d38882 in OpenVR::EventGfxThreadCallback(UnityVREventGfxThreadType, unsigned int, UnityVRGraphicsContext const*) ()
   from /home/Steam/Steam/steamapps/common/NeosVR/UnityPlayer.so
#16 0x00007fc6a048fbbc in GfxDeviceWorker::RunCommand(ThreadedStreamBuffer&) ()
   from /home/Steam/Steam/steamapps/common/NeosVR/UnityPlayer.so
#17 0x00007fc6a0493d6b in GfxDeviceWorker::RunExt(ThreadedStreamBuffer&) ()
   from /home/Steam/Steam/steamapps/common/NeosVR/UnityPlayer.so
#18 0x00007fc6a0489955 in GfxDeviceWorker::RunGfxDeviceWorker(void*) ()
   from /home/Steam/Steam/steamapps/common/NeosVR/UnityPlayer.so
#19 0x00007fc6a088601a in Thread::RunThreadWrapper(void*) ()
   from /home/Steam/Steam/steamapps/common/NeosVR/UnityPlayer.so
#20 0x00007fc69fc3efa3 in start_thread (arg=<optimized out>)
    at pthread_create.c:486
#21 0x00007fc69fb6f4cf in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Again, similar kernel log entries. I'm told that it's to do with fences not being signalled.

cirk2 commented 4 years ago

Got the same crash in vrwebhelper with the log output

[drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* syncobj 17 failed to find fence @ 0 (-22)!
[drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Failed in the dependencies handling -22!

It's happening with and without runtime enabled.

systemd-coredump:

 Process 100939 (vrwebhelper) of user 1000 dumped core.
 Stack trace of thread 100939:
#0  0x00007f79d6d45ce5 raise (libc.so.6 + 0x3bce5)
#1  0x00007f79d6d2f857 abort (libc.so.6 + 0x25857)
#2  0x00007f78c60489eb n/a (libvulkan_radeon.so + 0xcb9eb)
#3  0x00007f78c6048e5e n/a (libvulkan_radeon.so + 0xcbe5e)
#4  0x00007f78c604ce2f n/a (libvulkan_radeon.so + 0xcfe2f)
#5  0x00007f79d1a22072 n/a (/home/felix/.local/share/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so + 0x11a072)
#6  0x00007f79d1a2373d n/a (/home/felix/.local/share/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so + 0x11b73d)
#7  0x00007f79d1a4bd4b n/a (/home/felix/.local/share/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so + 0x143d4b)
#8  0x0000000000418b03 n/a (/home/felix/.local/share/Steam/steamapps/common/SteamVR/bin/vrwebhelper/linux64/vrwebhelper + 0x18b03)
#9  0x0000000000418d11 n/a (/home/felix/.local/share/Steam/steamapps/common/SteamVR/bin/vrwebhelper/linux64/vrwebhelper + 0x18d11)
#10 0x000000000051afd2 n/a (/home/felix/.local/share/Steam/steamapps/common/SteamVR/bin/vrwebhelper/linux64/vrwebhelper + 0x11afd2)

Running the executeable with gdb to bet a more detailed backtrace:

#1  0x00007fffee855857 in abort () from /usr/lib/libc.so.6
#2  0x00007ffed19ae9eb in ?? () from /usr/lib/libvulkan_radeon.so
#3  0x00007ffed19aee5e in ?? () from /usr/lib/libvulkan_radeon.so
#4  0x00007ffed19b2e2f in ?? () from /usr/lib/libvulkan_radeon.so
#5  0x00007fffeb538072 in CVRCompositorSharedTextures::EndCommandBuffer(CVRCompositorSharedTextures::CommandBuffer_t**) ()
   from /home/felix/.local/share/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so
#6  0x00007fffeb53973d in CVRCompositorSharedTextures::UpdateTextureSet(CVRCompositorSharedTextures::Params const*, VRCompositorState_TextureSet_t*) ()
   from /home/felix/.local/share/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so
#7  0x00007fffeb561d4b in CVROverlayLatest::SetOverlayTexture(unsigned long, vr::Texture_t const*) ()
   from /home/felix/.local/share/Steam/steamapps/common/SteamVR/bin/linux64/vrclient.so
#8  0x0000000000418b03 in CCefClientHandler::OnPaint(scoped_refptr<CefBrowser>, cef_paint_element_type_t, std::vector<CefRect, std::allocator<CefRect> > const&, void const*, int, int) ()
#9  0x0000000000418d11 in non-virtual thunk to CCefClientHandler::OnPaint(scoped_refptr<CefBrowser>, cef_paint_element_type_t, std::vector<CefRect, std::allocator<CefRect> > const&, void const*, int, int) ()
#10 0x000000000051afd2 in (anonymous namespace)::render_handler_on_paint(_cef_render_handler_t*, _cef_browser_t*, cef_paint_element_type_t, unsigned long, _cef_rect_t const*, void const*, int, int) ()
#11 0x00007ffff15dc875 in OnPaint () at ../../cef/libcef_dll/ctocpp/render_handler_ctocpp.cc:199
#12 0x00007ffff356b1c1 in OnPaint () at ../../cef/libcef/browser/osr/render_widget_host_view_osr.cc:1472
#13 0x00007ffff356cda3 in OnCopyFrameCaptureSuccess () at ../../cef/libcef/browser/osr/render_widget_host_view_osr.cc:255
#14 CopyFromCompositingSurfaceHasResult () at ../../cef/libcef/browser/osr/render_widget_host_view_osr.cc:233
#15 0x00007ffff356ce74 in Invoke<void (CefCopyFrameGenerator::*)(gfx::Rect const&, std::__1::unique_ptr<viz::CopyOutputResult, std::__1::default_delete<viz::CopyOutputResult> >), base::WeakPtr<CefCopyFrameGenerator> const&, gfx::Rect const&, std::__1::unique_ptr<viz::CopyOutputResult, std::__1::default_delete<viz::CopyOutputResult> > > () at ../../base/bind_internal.h:516
#16 MakeItSo<void (CefCopyFrameGenerator::* const&)(gfx::Rect const&, std::__1::unique_ptr<viz::CopyOutputResult, std::__1::default_delete<viz::CopyOutputResult> >), base::WeakPtr<CefCopyFrameGenerator> const&, gfx::Rect const&, std::__1::unique_ptr<viz::CopyOutputResult, std::__1::default_delete<viz::CopyOutputResult> > > () at ../../base/bind_internal.h:636
#17 RunImpl<void (CefCopyFrameGenerator::* const&)(gfx::Rect const&, std::__1::unique_ptr<viz::CopyOutputResult, std::__1::default_delete<viz::CopyOutputResult> >), std::__1::tuple<base::WeakPtr<CefCopyFrameGenerator>, gfx::Rect> const&, 0, 1> () at ../../base/bind_internal.h:689
#18 Run () at ../../base/bind_internal.h:671
#19 0x00007ffff236d522 in Run () at ../../base/callback.h:99
#20 Invoke<base::OnceCallback<void (std::__1::unique_ptr<media::VideoCaptureDevice, std::__1::default_delete<media::VideoCaptureDevice> >)>, std::__1::unique_ptr<media::VideoCaptureDevice, std::__1::default_delete<media::VideoCaptureDevice> > > () at ../../base/bind_internal.h:577
#21 MakeItSo<base::OnceCallback<void (std::__1::unique_ptr<media::VideoCaptureDevice, std::__1::default_delete<media::VideoCaptureDevice> >)>, std::__1::unique_ptr<media::VideoCaptureDevice, std::__1::default_delete<media::VideoCaptureDevice> > > () at ../../base/bind_internal.h:616
#22 RunImpl<base::OnceCallback<void (std::__1::unique_ptr<media::VideoCaptureDevice, std::__1::default_delete<media::VideoCaptureDevice> >)>, std::__1::tuple<std::__1::unique_ptr<media::VideoCaptureDevice, std::__1::default_delete<media::VideoCaptureDevice> > >, 0> () at ../../base/bind_internal.h:689
#23 RunOnce () at ../../base/bind_internal.h:658
#24 0x00007ffff3648e6b in Run () at ../../base/callback.h:99
#25 RunTask () at ./../../base/debug/task_annotator.cc:99
#26 0x00007ffff366281b in RunTask () at ./../../base/message_loop/message_loop.cc:434
#27 0x00007ffff3662ceb in DeferOrRunPendingTask () at ./../../base/message_loop/message_loop.cc:445
#28 DoWork () at ./../../base/message_loop/message_loop.cc:517
#29 0x00007ffff36642d9 in base::MessagePumpGlib::Run(base::MessagePump::Delegate*) () at ./../../base/message_loop/message_pump_glib.cc:309
#30 0x00007ffff36818d5 in Run () at ./../../base/run_loop.cc:102
#31 0x00007ffff3526a50 in CefRunMessageLoop () at ../../cef/libcef/browser/context.cc:306
#32 0x00000000004294f7 in RealMain(int, char**) ()
#33 0x000000000042aee2 in int VRCrashHandlerWrapMain<int, char**, int&, char**&, int>(int (*)(int, char**), int&, char**&) ()
#34 0x000000000042966a in main ()

System Information (please complete the following information): Distribution: Arch SteamVR version: 1.10.32 (beta branch) Steam client version: 1585009313 Opted into Steam client beta?: Yes Graphics driver version: Mesa git 43918c9a7fc (2020-03-24) GPU: Vega 64 SteamVr System Report

dsalt commented 4 years ago

Curiously, vrwebhelper seems to work fine with amdvlk-open (I'm using a fresh local build, possibly slightly post-dating 2020.Q1.4), but I then have to make the games use that too else I get random fixed-ish junk displayed by the HMD. [Edit: looks like that was temporary.]

I altered SteamVR/bin/vrwebhelper/linux64/vrwebhelper.sh to export VK_ICD_FILENAMES, pointing at the installed amd_icd64.json for amdvlk-open.

The settings window works fine in this configuration, but I don't currently have a working dashboard – it's clearly there, but just not visible.

This also seems to be enough to get Half-Life: Alyx working.

dsalt commented 4 years ago

See also https://gitlab.freedesktop.org/mesa/mesa/-/issues/2705

cirk2 commented 4 years ago

Yeah that workaround works for me too

farmboy0 commented 4 years ago

I suddenly got the same error and I could fix it by unsettring mesa_glthread=true, i had it set globally so it affected HL:A and SteamVR.

im-0 commented 4 years ago

Same issue. unset mesa_glthread before launching Steam fixes this.

System Information (please complete the following information):

dsalt commented 4 years ago

Just built some fresh Mesa .debs (along with libdrm and libglvnd) and tested without AMDVLK. All seems well so far.

Changes since initial report:

im-0 commented 4 years ago

Yes, this was fixed by https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5600 (which is in Mesa 20.2). I confirm.

kisak-valve commented 4 years ago

Closing per the last couple comments.