GPU memory leak with HEVC hardware decoder when draining and sending MFT_MESSAGE_NOTIFY_END_STREAMING message

[bluetarpmedia]

There appears to be a leak with GPU dedicated memory for some HEVC video streams when using the HEVC hardware decoder and sending the MFT_MESSAGE_COMMAND_DRAIN message followed by the MFT_MESSAGE_NOTIFY_END_STREAMING message.

Reproduced on:

• Windows 11, AMD Radeon RX 5700 XT
• Windows 11, NVIDIA GeForce RTX 4070 Laptop GPU

Steps to reproduce:

1. Build the attached sample code twice, once with sending the MFT_MESSAGE_NOTIFY_END_STREAMING message enabled (line 510) and another time with it disabled.
2. Run the program and observe the GPU dedicated memory usage.
3. The program requires an HEVC encoded sample (attached). This is the first frame (an IDR sample) from an HEVC video stream.

hevc10bit.cpp.txt sample.zip

The sample code has a loop (the iterations can be adjusted with the RepeatCount variable) with these steps:

1. Push HEVC IDR sample to decoder
2. Send Drain message
3. Process output, confirm that it retrieves the decoded frame
4. Send the End Streaming message (optional: test with / without this step)
5. Send the Flush message

When sending MFT_MESSAGE_NOTIFY_END_STREAMING we observe growth in GPU memory usage that grows unbounded for as long as the test runs:

When we don't send MFT_MESSAGE_NOTIFY_END_STREAMING we observe flat GPU dedicated memory usage:

[RohitAthavale]

Hello bluetarpmedia,

Calling ProcessMessage with MFT_MESSAGE_NOTIFY_END_STREAMING followed by MFT_MESSAGE_COMMAND_FLUSH does not follow the order of calling ProcessMessage while tearing down the MFT described in the Basic MFT Model .

Could you try by following steps 4 - 8 in the Process Data section within the processing loop of the file you had linked.

[bluetarpmedia]

Thanks for your reply @RohitAthavale. If we remove END_STREAMING and insert END_OF_STREAM before the DRAIN as suggested in steps 4-8 then there is no leak. (Likewise there is no leak simply by removing END_STREAMING; that is to say, there is no need to add END_OF_STREAM to avoid the leak.)

We don't need the ability to send either END_STREAMING or END_OF_STREAM messages. But we noticed that sending END_STREAMING after the drain and before the flush causes this memory leak with the HEVC hardware decoder with some HEVC video streams, and also does not happen with the AVC hardware decoder, which made it appear to be a bug.