search

microsoft / media-foundation

Repository for Windows Media Foundation related tools and samples
MIT License
144 stars 31 forks source link

Crash when using VP8 decoder with D3D11 #48

Closed robUx4 closed 1 year ago

robUx4 commented 1 year ago

I have enabled VP8/VP9 decoding through MediaFoundation in VLC. When using the software decoder everything works fine. But If I enable the D3D manager, it crashes when I select the NV12 output.

Here is the pseudo code:

It crashes with the following logs:

D3D11 ERROR: ID3D11Device::CheckFeatureSupport: Returning E_INVALIDARG, meaning the parameters were invalid. [ STATE_GETTING ERROR #2097315: DEVICE_CHECKFEATURESUPPORT_INVALIDARG_RETURN]
Exception thrown at 0x00007FFEA0D6CD29 in vlc.exe: Microsoft C++ exception: _com_error at memory location 0x000000B7E55FDC00.
D3D11 ERROR: ID3D11DeviceContext::CreateVideoDecoder: Invalid DXGI_FORMAT 67 specified, failing! [ STATE_CREATION ERROR #3145760: CREATEVIDEODECODER_INVALIDFORMAT]
Exception thrown at 0x00007FFEA0D6CD29 in vlc.exe: Microsoft C++ exception: _com_error at memory location 0x000000B7E55FDCF0.
Exception thrown at 0x00007FFEA0D6CD29 in vlc.exe: Microsoft C++ exception: _com_error at memory location 0x000000B7E55FDC00.
D3D11 ERROR: ID3D11DeviceContext::CreateVideoDecoder: Invalid DXGI_FORMAT 67 specified, failing! [ STATE_CREATION ERROR #3145760: CREATEVIDEODECODER_INVALIDFORMAT]
Exception thrown at 0x00007FFEA0D6CD29 in vlc.exe: Microsoft C++ exception: _com_error at memory location 0x000000B7E55FDCF0.
Exception thrown at 0x00007FFE1071B338 (msvp9dec_store.dll) in vlc.exe: 0xC0000005: Access violation reading location 0x0000000000000000.

output_media_type has a MF_MT_SUBTYPE of MFVideoFormat_NV12

This is with an NVIDIA 3070 with drivers 31.0.15.3141 on Win10 19045.2728

If I don't do this part it works fine (but it will never use the hardware decoder which the GPU supports):

The same code works to decode H264 with the hardware decoder and D3D11 output (using SetInputType(MFMediaType_Video.MFVideoFormat_h264))

robUx4 commented 1 year ago

If I call IMFTtransform::SetOutputType() with MFT_SET_TYPE_TEST_ONLY, there is no crash and it returns S_OK.

The code doesn't crash if I don't call IMFDXGIDeviceManager::ResetDevice() (to use the D3D11Device tied to my display, which is the default D3D11Device).

If I use an Intel UHD Graphics 630 GPU it doesn't crash. That GPU supports VP8 hardware decoding, unlike the NVIDIA RTX 3070.

cjrog commented 1 year ago

Looks like there was an access violation in the VP9 decoder MFT. Can you share a Heap Dump from the failure to help us investigate?

robUx4 commented 1 year ago

Here is a minidump with heap when the crash occurs. (I can't upload a file bigger than 25 MB here)

https://www.icloud.com/iclouddrive/0f2WWjkYGZjtj7h6KHy3-W9Sw#vp8-mft-crash

Here is a smaller one (less DLLs loaded) https://www.icloud.com/iclouddrive/02fQtJLm755FtuseZht7Ho7Dw#vp8-mft-crash2

DavidS-msft commented 1 year ago

Thanks, we're looking into the dump provided.

mattrwoz commented 1 year ago

Yeah, we confirmed that this crash is definitely a bug in the decoder. I have opened a bug and should be able to get it fixed pretty easily. Though it is crashing in a failure path. So even with the fix the decoder is going to fail out and return an error code instead of crashing.

Did the call to IMFTtransform::ProcessMessage(MFT_MESSAGE_SET_D3D_MANAGER) succeed? That call is what specifically checks if the hardware supports acceleration, and should fail if not. Most hardware should support VP9 acceleration, but there are still some systems out there that don't.

Also, when creating the D3D11 device to be used for decode make sure to specify D3D11_CREATE_DEVICE_VIDEO_SUPPORT in the feature flags.

robUx4 commented 1 year ago

Did the call to IMFTtransform::ProcessMessage(MFT_MESSAGE_SET_D3D_MANAGER) succeed?

Yes.

Most hardware should support VP9 acceleration, but there are still some systems out there that don't.

This is a VP8 stream. Some Intel GPUs can decode it, but not the NVIDIA RTX 3070 (as reported by DXVA Checker).

Also, when creating the D3D11 device to be used for decode make sure to specify D3D11_CREATE_DEVICE_VIDEO_SUPPORT in the feature flags.

I'll had a check, but our internal ID3D11Device always has this flag set. And it was set in this case.

mattrwoz commented 1 year ago

Thanks for checking. I suspect there might be a bug in ProcessMessage for VP8 then. It should have failed in this case; it does have logic to fail if VP8 is configured and there is no acceleration support. I will take a look and try to bundle both fixes into the next release (usually takes about 1 month for a codec update to be fully deployed).

mattrwoz commented 1 year ago

We have successfully tested a version of the VP9 decoder which avoids the crash, and is able to correctly report that VP8 is not supported on this hardware when calling SetInputType() with a VP8 subtype. As long as testing goes well, starting in the next release the codec will report MF_E_UNSUPPORTED_D3D_TYPE when VP8 is not supported by hardware acceleration.

(The codec is updated on all supported OSes down to Win10, and almost all users will get updated. If necessary, I can provide codec version information once we start release if you want to be absolutely sure you are using a good version of the codec (it is possible to check which version is used programmatically)).

robUx4 commented 1 year ago

Excellent news! Thanks for the fix. I'll be monitoring this.