Memory leak in hooked application

[SladeThe]

I've built an application based on Direct3DHook. It makes frequent screenshots of a DirectX 11 game. It works well for a short duration, but the RAM usage of this game grows quickly over time. The more often I take screenshots, the quicker this game eats all available system memory and crashes. The RAM usage of my managed application is normal.

I create CaptureProcess with default settings (once)

CaptureConfig captureConfig = new CaptureConfig();
CaptureInterface captureInterface = new CaptureInterface();
CaptureProcess captureProcess = new CaptureProcess(process, captureConfig, captureInterface);

and then use it this way (many times)

using (Capture.Interface.Screenshot screenshot = captureProcess.CaptureInterface.GetScreenshot(
    Rectangle.Empty, new TimeSpan(0, 0, 5), null, Capture.Interface.ImageFormat.Bitmap
)) {
    ...
}

[justinstenning]

Can you please try with Png image format, and PixelData to see if they also exhibit the behaviour.

[SladeThe]

Disabled all my code inside using and re-checked.

Both Bitmap and Png formats cause memory leak. Probably, Png does it slower, but I'm not sure.

PixelData does not. However, I need to improve my code to be able to use it.

[justinstenning]

Ok thanks, not sure what changed to introduce that, fairly certain it was good for a while. I’ll try to take a look over the next month.

[SladeThe]

I'm not very experienced in low level graphics programming, but I made some investigations with dotTrace. There is a leak of unmanaged memory allocated in SharpDX.WIC.Bitmap's constructor inside DXHookD3D11.ToStream.

I'll try to fix it by myself and will make a PR in case of success. Otherwise, I hope that my investigation will help you to find the problem and to fix it.

[SladeThe]

Further investigations show that the converter.Initialize(bitmap, ...) is the cause of memory leak. When this call happens, a bitmap can not Dispose in a right way.

Disabled converter temporarily in my project, but I'm sure you can find a more elegant solution.

[justinstenning]

@SladeThe yeah, I have missed wrapping the converter in a using clause in a recent update e.g. var converter = new SharpDX.WIC.FormatConverter(wicFactory); becomes:

using (var converter = new SharpDX.WIC.FormatConverter(wicFactory)) 
{
    ...
}

[SladeThe]

Yes. That helped, thanks. But doesn't really matter.

The following code works fine for me. Both with and without converter.

using (MemoryStream dataStream = new MemoryStream(screenshot.Data)) {
    using (Bitmap bitmap = (Bitmap) Image.FromStream(dataStream)) {