microsoft / DockerTools

Tools For Docker, including Visual Studio Provisioning and Publishing
Other
173 stars 26 forks source link

Debugging services ran out of memory #357

Open clintsinger opened 1 year ago

clintsinger commented 1 year ago

When I'm doing long running tests I'm am very likely to run into the following error:

image

This seems to happen based on the amount of logging output I'm generating. i.e if I'm outputting a lot of logging I will encounter it sooner than if I am not putting out as much logging.

Once this has occurs I need to exit visual studio and have all of the docker containers recreated to start debugging again.

I'm using a project with many containers being orchestrated by docker compose.

NCarlsonMSFT commented 1 year ago

Sorry my response to this appears to have not posted. A couple of questions: Which version of Visual Studio are you using? Are you using windows or Linux containers? If Linux, are you using the Hyper-V Docker host or WSL 2? Have you been able to determine if the out of memory issue is specific to a debuggee running in container, or is VS running out of memory on the host?

clintsinger commented 1 year ago

Visual Studio: 17.3.4 Docker: 20.10.17

The containers are Linux containers in WSL2

It feels like the issue is the debuggee is running out of memory. It seems like the issue is caused when there has been a large quantity of logging. I say this because the system is able to run much longer when I remove the logging. I don't think there is any kind of purging of the docker logs which is why it requires shutting them down (typically done by closing VS). If it is possible for the container tools to clear a percentage of the head of the log that may help.

gregg-miskelly commented 1 year ago

I found and fixed the issue in vsdbg (the .NET debugger). We should hopefully have a release out soon with the fix.

danegsta commented 1 year ago

The .NET debugger should now be updated with a fix for this issue; you can ensure the VS container tooling is using the latest version by deleting the debugger cache folder at %userprofile%\vsdbg and then opening your solution and starting a debug session. Let us know if you're still seeing any issues after updating.

clintsinger commented 1 year ago

I cleared the cache and started up my projects but after a few hours I now encountered the following image

It is a different dialog, as it doesn't call out memory specifically so it may be a new issue or still related to this one. I'll have to run it a few more times to monitor. Though it is a slow process because of how long it takes between failure. I'll get back to you when I have more information.

gregg-miskelly commented 1 year ago

That failure is probably unrelated. It is CORDBG_E_TARGET_INCONSISTENT (The debuggee is in a corrupt state) which is coming from the .NET Runtime in the target process.

The .NET Runtime raises this error in a bunch of states, so it is hard to say more without additional information. https://github.com/dotnet/runtime/search?q=CORDBG_E_TARGET_INCONSISTENT

clintsinger commented 1 year ago

So far things are looking good. I'll continue to test as I haven't been able to have a situation where I have been able to leave it running for a very long time. It does manage to run overnight and before it would usually be dead by the morning.