Virtual DMD crashes if Close() is called at any point

[mjrgh]

This is closely related to my pull request about the PM_GameSettings crash, but a fix has eluded me so far, so I'm just going to file a bug report for now. Maybe I'll try to figure out what's going on later if I have the time, but hopefully you'll have more insight into the problem and will beat me to it. :)

If the virtual DMD is open, and the client calls Close() at any point, and then re-opens the session by either calling Open() directly or just calling any other function (since everything implicitly calls Init()), the process will crash. This sometimes manifests as getting wedged (i.e., all threads are blocked, program is frozen), and sometimes outright crashes with a hard exception down in some NtXxx kernel layer.

The root of the problem is that Close() isn't fully cleaning up the session. It's leaving something related to the dispatch thread alive, and in fact I think it's just leaving the dispatch thread alive. So if we ever go back through Init(), that's where things go south - Init() tries to get that dispatch thread going again, which either gets into a thread deadlock or hits that kernel crash. I've tried a bunch of things to try to get the dispatch thread to well and truly exit so that we can get back to the initial state, but I can't find anything that works. Hopefully you have a better idea of how the threading is supposed to work and can figure out how to kill the thread dead so that we can safely reinitialize if necessary.

That pull request I put in yesterday is just another facet of this - it was crashing due to exactly the same root problem. This is just another route into it.

As a workaround, I'm just avoiding all calls to Close(). But that's not great because it leaves the virtual DMD on the screen if it was ever shown.

[mjrgh]

I suppose one easy workaround would be to just stop doing the partial cleanup that Close() is doing, leaving everything fully initialized, but hiding (not closing) the virtual DMD window. That would at least look to the user like a proper Close() was happening. Init() would then just re-show the window (and do nothing else) if things are already running.

[freezy]

Well, the dmddevice.dll API was created for real DMDs, so the close semantic means "clean up and free any resources you might be using because we're done here". Thus I would rather fix the crashes than change the semantic.

Do you have a stack where it crashes?

[mjrgh]

Yeah, it'd certainly be better to really clean up. Unfortunately, the stack trace I get is useless. It's down in the kernel, so it just has a couple of internal NtHandleException type levels and nothing more; it's one of those contexts where the debugger can't trace the stack back into user-mode code because there aren't normal C stack frames to trace. The thread context is a CLR internal thread.

[mjrgh]

It should be easy to reproduce, though - set up for virtual dmd, call Open(), write a frame, call Close(), call Open(), write another frame.

[freezy]

Alright, I'll see what I can do!