Using SOS programmatically

[raffaeler]

Hello, I would like to automate some diagnostics by programmatically issuing commands and getting back the results. From the codebase I see the final string is created at SOS level and not in dotnet-dump. I would like to avoid parsing the text and hoping in an easy way to get the results from the commands before they get formatted in strings.

I know another way is going straight with CLRMD, but SOS is already doing what I need, and I want to just adding an automatic processing.

[mikem8361]

We don't recommend trying to use the existing SOS commands programmatically. Running the SOS command and "scrapping" the text isn't a reliable option. Like you said, going straight to ClrMD is what we recommend. In the future, we are going to make it easier to load your own ClrMD based commands in dotnet-dump.

[raffaeler]

I understand the point, but I would really be able to add some automation to the tool, without having to mess with the CLRMD entirely (which is a thing that I already did but it can definitely discourage a lot of users).

For example, in order to find the statics, I have to loop through all the objects and find the address matching the root. Since WinDbg and lldb have a totally different set of commands, I would prefer the ability to "extend" dotnet-dump instead.

[davidfowl]

I think it would be interesting to run the sis commands non interactively. It might need some more thoughts though since it’s useful to run a bunch of commands to then pipe to output

[raffaeler]

@davidfowl I just did it in a small sample for a conference. The bad thing of this small demo is that I remote control the dotnet-dump via stdin/stdout (and I am aware it may fail in future versions). The good thing is that I created a queue of commands that are run asynchronously to loop over a large number of objects. The result is that I can finally find the static roots responsible for my context being kept in memory :)

I just wish the dotnet-dump may become extensible for making a specific diagnostic easy to implement.

[mikem8361]

I'm working on a design doc about dotnet-dump extensibility. I'll send out a link with it is ready.

[raffaeler]

@mikem8361 One of the possible way to extend the tool is using stdin/stdout as most of the tools in the Linux world do. But this would require to support "templating" like the awewome docker CLI does. Substantially, you pass a template to the CLI so that the output is formatted according to the template and the output can be easily retrieved by other tools following in the pipe.

Said that, as a Windows developer I love the extensibility using CLRMD, but the result you get from that library is somewhat different from the SOS command, therefore it takes a high amount of time reaching the same goal. Also, going to Linux is definitely not trivial as it requires to emulate COM.

Thanks

[mikem8361]

The extensibility I'm talking is mostly about ClrMD based commands. The ability to load assemblies containing these commands in dotnet-dump, lldb and cdb/windbg.

The existing native SOS commands would really be hard to "template" or make more extensible. The direction would be to (slowly because of limited resources) to convert/rewrite the commands using ClrMD using this extensiblity model. We are even hesitant to provide an execute command API because of all the problems seen in dbgeng extensions with screen scraping text, etc.

/cc: @noahfalk

[raffaeler]

I recently worked in the COM emulation and open to provide help on this topic as well as other debugging features I recently implemented on both Windows and Linux. (Noah knows it).

On the tool perspective, IMO the important thing is to have parity between the API-based and the SOS commands output. Of course I expect to have more power in my hand with the APIs.

[noahfalk]

I just wish the dotnet-dump may become extensible for making a specific diagnostic easy to implement

@raffaeler - In your ideal world would the extensibility primarily be about being integrated dotnet-dump's UI, or getting access to dotnet-dump's internal APIs (perhaps for use with a different UI), or both simultaneously?

[raffaeler]

Hi @noahfalk , the internal dotnet-dump apparently doesn't have any interesting starting point for the extensibility as all the data is managed in native cpp code formatting the strings. On the other side, the CLRMD library isn't intuitive for anybody to build any domain-specific tool and not easy to move across platforms.

To be clear: CLRMD is very good, but In my experience, there will be no many customers willing to study it to develop an internal tool. IMO it should be more friendly to make the community diagnostic ecosystem grow.

My ideal solution would be to consolidate CLRMD to be truly usable cross-platform and offering the same commands that SOS is exposing now, but with data (no strings). After that, dotnet-dump should be turned into a mere front-end printing the results on the console.

This should make easier to extend the SOS commands, plug-in additional commands, build custom searches, build other UIs (also graphical), and finally tools being able to pipe the results on Linux as well.

As a side note I opened an issue in the CLRMD repo to identify the owning LoadContexts for the objects.

[noahfalk]

Thanks for the feedback!

but In my experience, there will be no many customers willing to study it to develop an internal tool

Of course numbers are relative and we can always do better but we've historically considered it a success in terms of how many people use it : ) On NuGet it has 921K downloads and we hear from people in the community using it.

My ideal solution would be to consolidate CLRMD to be truly usable cross-platform and offering the same commands that SOS is exposing now, but with data (no strings)

That aligns pretty well with discussions the team has had internally, with the exception that maybe that SOS-like functionality is in a library on top of CLRMD instead of CLRMD itself. My main challenge is around priorities as it appears like a fair amount of work to rewrite all that C++ SOS code into C# with nicely factored separation between the View and Model. I also don't hear about this issue much from users. Not sure if you have any interest turning your powerhouse ICorDebug prowess towards some SOS PRs? : )

[raffaeler]

Hi @noahfalk You are more than welcome!

Of course numbers are relative and we can always do better but we've historically considered it a success in terms of how many people use it : ) On NuGet it has 921K downloads and we hear from people in the community using it.

Agree, my fault, I was not clear. I meant that the adoption could be far higher if the extensibility is easier.

maybe that SOS-like functionality is in a library on top of CLRMD instead of CLRMD itself.

It definitely makes sense

it appears like a fair amount of work to rewrite all that C++ SOS code into C#

I know, but I strongly believe that it would be a great plus for the entire ecosystem.

Not sure if you have any interest turning your powerhouse ICorDebug prowess towards some SOS PRs

I am more than willing, and you know :) I can eventually write from scratch a new different version of the interop generation and publish it. It probably could take some more time, but I would like to contribute on this.

[noahfalk]

Thanks @raffaeler !

I can eventually write from scratch a new different version of the interop generation and publish it

Sorry I may have muddied the water a bit, I wasn't referring to the specific interop technology in this case but rather your tenacity and effectiveness at the project :) If SOS's commands are going to build on CLRMD then I'd imagine that we'd be using pre-existing wrappers implemented inside CLRMD to facilitate the native<->managed interop.

I'm probably starting to get ahead of myself a bit, @mikem8361 is our main SOS guru so I want to make sure anything we do is aligning with his plans. The other part as mentioned is that general SOS extensibility probably won't be a high priority on the team at MS so you will likely see us being measured with investments. I'd be reticent to make a big commitment of time or to substantially extend the set of features/public APIs/NuGet packages Microsoft runtime devs become responsible for maintaining. Within those limits though if we can help the community to get closer to tooling nirvana I'd love to do so.

[raffaeler]

You are welcome @noahfalk :) I totally understand that this is not a top priority and I am willing to collaborate with you both to make the debugging experience better. I am not sure how much I can help, but will try to stay tuned and see :)

Please, let's stay in touch so that I can follow the evolutions on the project.