search

baldurk / renderdoc

RenderDoc is a stand-alone graphics debugging tool.
https://renderdoc.org
MIT License
8.6k stars 1.3k forks source link

AccessViolationException when using ICompositorInterop::CreateGraphicsDevice on Windows #3342

Closed jp2masa closed 4 weeks ago

jp2masa commented 1 month ago

~EDIT: If you are affected by this issue, please vote here: https://developercommunity.visualstudio.com/t/Access-violation-when-using-RenderDoc-wi/10677651.~

EDIT 2: If you are affected by this issue, please vote here: https://aka.ms/AAqsqws.

Description

If an app uses ICompositorInterop::CreateGraphicsDevice, debugging it with RenderDoc causes it to throw an AccessViolationException.

This is related to #2923, which was supposed to be fixed by blocking the internal interfaces. However, instead of failing, CreateGraphicsDevice causes an AccessViolationException. Maybe there's some new interface being queried?

Steps to reproduce

It can be reproduced using an empty Avalonia application (https://github.com/AvaloniaUI/Avalonia/issues/15911). However, trying to create a graphics device using ICompositorInterop::CreateGraphicsDevice should be enough (something like https://github.com/AvaloniaUI/Avalonia/blob/83d7c25023b4ccdd1dedf5725f4e91f6f849de90/src/Windows/Avalonia.Win32/WinRT/Composition/WinUiCompositedWindowSurface.cs#L81-L84).

Environment

baldurk commented 4 weeks ago

This looks to be a bug inside DirectComposition. It's not checking the return value from querying the banned interface and unconditionally using the returned pointer without any kind of NULL check or error handling, so there's nothing I can do about this in RenderDoc.

jp2masa commented 4 weeks ago

Thanks anyway. It's sad to see so many problems with DirectComposition... I'll see if I can report this issue to them, maybe it will be fixed someday.