"FATAL ERROR: CreateWindowEx failed" with GarchGLDebugWindow

ennis commented 5 years ago

Description of Issue

A simple application using GarchGLDebugWindow crashes on windows with error

FATAL ERROR: CreateWindowEx failed
in Init at line 93 of usd\pxr\imaging\lib\garch\glplatformdebugwindowwindows.cpp

Steps to Reproduce

Create and Initialize a GarchGLDebugWindow like so:

int main(int argc, char** argv)
{
pxr::GarchGLDebugWindow window("Hello World", 1280, 720);
window.Init();
window.Run();
return 0;
}

System Information (OS, Hardware)

OS: Windows x64

Package Versions

USD v19.05

Build Flags

Built USD with --debug

Additional Information

There seems to be several issues with the code in glPlatformDebugWindowWindows.cpp:

When calling CreateWindowEx, the message handler &Garch_GLPlatformDebugWindow::_MsgProc is called but the window has not been registered yet in _GetWindowsMap. In this case, the message handler returns 0 instead of forwarding to DefWindowProc. https://github.com/PixarAnimationStudios/USD/blob/32ca7df94c83ae19e6fd38f7928d07f0e4cf5040/pxr/imaging/lib/garch/glPlatformDebugWindowWindows.cpp#L168-L170
There is a typo (_hInstnace) in Garch_GLPlatformDebugWindow::Init. The rest of the code uses the uninitialized value of the member variable _hInstance. https://github.com/PixarAnimationStudios/USD/blob/32ca7df94c83ae19e6fd38f7928d07f0e4cf5040/pxr/imaging/lib/garch/glPlatformDebugWindowWindows.cpp#L67
The constructor of Garch_GLPlatformDebugWindow does not set _callback, causing the program to crash when it is invoked here: https://github.com/PixarAnimationStudios/USD/blob/32ca7df94c83ae19e6fd38f7928d07f0e4cf5040/pxr/imaging/lib/garch/glPlatformDebugWindowWindows.cpp#L130

After fixing all of these, the window initializes seemingly without error.

c64kernal commented 5 years ago

Thanks very much for catching these, @ennis. Feel free to submit a PR if you can. If you can't (e.g., because of CLA issues), let us know. Thanks again!

jilliene commented 5 years ago

Filed as internal issue #USD-5385

lightwolf commented 4 years ago

This is my attempt on fixing the issue: #1102

PixarAnimationStudios / OpenUSD