microsoft / DbgShell

A PowerShell front-end for the Windows debugger engine.
MIT License
674 stars 89 forks source link

Q: Is Shell the only option? What about a module? #20

Closed nightroman closed 6 years ago

nightroman commented 6 years ago

The approach with DbgShell.exe is clear. But the console UX is somewhat poor. There are other hosts, VSCode + PowerShell, to start with, and some more, e.g. my own Far Manager + PowerShellFar (I am interested in using it specifically). Other hosts provide some powerful tools, UI, etc. For example, Tab completion in DbgShell is better than nothing but drop down Intellisense menus in other hosts are much better. Or, say, other hosts allow debugging of PowerShell scripts designed for debugging of something else. Etc., etc.

In other words, is DbgShell.exe the only option or the normal PowerShell module package possible, too?

jazzdelightsme commented 6 years ago

Currently, the conhost-based DbgShell.exe (standalone or hosted in windbg) is the only option.

When I first started the project, I had the same idea as you—I wanted it to be a standard module that could be imported into any host. You can still see some vestiges of this—for instance, despite the fact that DbgShell implements its own Formatting+Output engine, there are also some standard PowerShell formatting definitions (.ps1xml), in Debugger.Format.ps1xml.in. Many of these are superseded by the DbgShell-native formatting definitions (in *.psfmt), but I think some may still be used (perhaps the ColorString view and some of the namespace objects?). Notice that my .ps1xml file is actually .ps1xml.in—it's actually a T4 template, and at build time, the Debugger.Format.Color.tt and Debugger.Format.tt files transform the template into a color-supporting version and non-color-supporting version.

(Side note about color support: Win10 conhost now supports most of the SGR codes used by DbgShell, but DbgShell still uses a couple that are not implemented by conhost (and conhost has not implemented them because they are non-standard).)

However, along the way, there were some difficulties. I don't think I will remember all of them off the top of my head... but in addition to difficulties with F+O and color (part of it may have been that I was just tired of creating and testing two formatting definitions for everything), there was a mismatch between dbgeng's global singleton model and PowerShell's many-runspaces-per-process model (not that it would be impossible to somehow block the Debugger module from being loaded into more than one runspace in a process). At some point I decided that I would just forget about the "module loadable anywhere" idea and just make DbgShell.exe work.

Of course you are welcome to look into what it would take to make it loadable into an alternate host; I just want to give heads up that there will be some challenges, even if I can't remember what they all are, and let you know that I am not currently planning on tracking this as a goal. Honestly, I'd probably be more interested in bringing cool stuff into the DbgShell.exe host (I've seen some pretty neat conhost-based UI, though I don't remember any project names at the moment).

There is also some frankly ridiculously convoluted code in the host that allows things to work when running from the !dbgshell extension in windbg; but other hosts wouldn't be hostable in windbg anyway, so I think you should be able to mostly sidestep that mess.

nightroman commented 6 years ago

Thank you for such a detailed answer. It's good to know that this is not impossible, at least in theory. It would be nice if this option is still on the radars even though I do not expect this any time soon. For the beta there are much more important things to do, I understand.

This was just a question and partially a suggestion for the future. Feel free to close it for now to avoid the noise in issues.

P.S. I mentioned some hosts that provide powerful and easy to use UI and tools. But this is just one part of the potential use cases. I can think of some UI-less completely automated debugging scenarios if it is possible to run the tools just in a PowerShell runspace, ideally with the default host, or at least with some specifically adapted.

jazzdelightsme commented 6 years ago

UI-less completely automated debugging scenarios

That's another thing I'd like to do, but I haven't had time to test it out—my idea is that you should be able to use the C# object model (DbgProvider.dll) and possibly also the DLR and be able to do programmatic things in C#. Since I haven't tried it out, I would expect there to be some snags (which could hopefully be ironed out).

For unattended scripting (actual powershell, not C#), I would envision something simple like creating a window-less DbgShell.exe process and pass it a script path on the command line. (Might be doable today, actually.)