Support only AV_PIX_FMT_D3D11 for d3d11

[geminixdev]

I implemented this in my App, and it crashes. Tested only on Ryzen 7 PC with AMD GPU. Tested with Qt 6.6.1, 6.6.2 and Latest 6.6.3 dev git, always the same.

The crash happens also with the video widget example.

Here is the output using the video widget example (fresh download today):

Creating hardware device context: d3d11va
Using hardware device context: d3d11va
mediaStatusChanged LoadedMedia PlayingState
Video streams: 1
[ 0 ] QMap(("handler_name", "VideoHandler")("language", "und")("rotate", "0")("vendor_id", "[0][0][0][0]")) 97500 frames, 0.04 frame rate
h264 : supported hardware device contexts:
    dxva2
    d3d11va
    cuda
    vaapi
    vulkan
Available pixel formats:
   yuv420p : AVPixelFormat( 0 )
   vulkan : AVPixelFormat( 191 )
   cuda : AVPixelFormat( 117 )
   dxva2_vld : AVPixelFormat( 51 )
   d3d11va_vld : AVPixelFormat( 116 )
   d3d11 : AVPixelFormat( 172 )
   vaapi : AVPixelFormat( 44 )
Using hardware decoding in d3d11
D3D11: Removing Device.
D3D11 ERROR: ID3D11Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware). [ EXECUTION ERROR #378: DEVICE_REMOVAL_PROCESS_AT_FAULT]
Failed to share FFmpeg texture: -2005270523 "The GPU device instance has been suspended. Use GetDeviceRemovedReason to determine the appropriate action."
Device loss detected in Present()
Failed to share FFmpeg texture: -2005270523 "The GPU device instance has been suspended. Use GetDeviceRemovedReason to determine the appropriate action."
Device loss detected in Present()
Failed to share FFmpeg texture: -2005270523 "The GPU device instance has been suspended. Use GetDeviceRemovedReason to determine the appropriate action."
Device loss detected in Present()
Failed to share FFmpeg texture: -2005270523 "The GPU device instance has been suspended. Use GetDeviceRemovedReason to determine the appropriate action."
Device loss detected in Present()
... (and so on until it crashes) ...

Reverting to the code before #453 has the effect that the crash does not happen, no error at all, very stable rendering (tested for hours and hours).

I understand that this somehow means that

• the code in src/QtAVPlayer/qavhwdevice_d3d11.cpp starting at line 172 is not executed, we return before reaching it.
• there is no zero copy rendering.
• The D3D11 texture as decoded is not used as is.
• More CPU / GPU is used than needed.

There have recently been many changes and fixes in Qt Multimedia code concerning textures and rendering, also for D3D11 decoding / rendering, to avoid crashes and to optimize rendering. This crash seems to have the effect that we cannot benefit from them yet.

[valbok]

@geminixdev

The crash happens also with the video widget example.

1. Is it reproducible for all/many video files?
2. Tested only on Ryzen 7 PC with AMD GPU. Is it possible to check for different PC?

I tested qml_video/widget_video examples for many files for 6.6.2 and 6.7.0, but Nvidia graphics.

3. Could you try also ffplay on the same PC?
4. Or QMediaPlayer, does it use D3D11 textures?

[valbok]

D3D11: Removing Device. D3D11 ERROR: ID3D11Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware). [ EXECUTION ERROR #378: DEVICE_REMOVAL_PROCESS_AT_FAULT]

5. Could you also confirm that Qt < 6 versions it works? If so, then might be an issue with renderers. If ffplay does not work, might be an issue inside ffmpeg. Or might be needed to upgrade DirectX?

[valbok]

6. Btw I use the latest from https://github.com/BtbN/FFmpeg-Builds/releases

[geminixdev]

6. Btw I use the latest from https://github.com/BtbN/FFmpeg-Builds/releases

Yes, same. It wasn't the very latest, but I'll update to latest from today and retest.

[geminixdev]

If ffplay does not work, might be an issue inside ffmpeg. Or might be needed to upgrade DirectX?

• ffplay works perfectly. (using ffplay from BtbN 6.1)
• windows 11, it is fully updated, so I assume latest DirectX too.

[geminixdev]

In the Qt bugreports there had been crashes reported with AMD GPUs, apparently there was or is an issue when you try to render ffmpeg d3d11 decoded textures directly. It seems to be fixed in latest Qt though.

This is why I tested all, from Qt 6.6.1 over Qt 6.6.2 to latest Qt 6.6 dev, the coming 6.6.3.

I had to setup a dev system recently, due to hardware crash, so currently I can develop only with Ryzen 7 and AMD GPU, and all old Qts are gone, too.

I might be able to test on an Intel i7 system in about a week.

[geminixdev]

1. Is it reproducible for all/many video files?

Any h264/aac mp4 or .ts file or stream I tried. The same I use all the time, whioch work fine everywhere, Quicktime, ffplay, VLC, or packed as HLS in HLS players.

[valbok]

It seems to be fixed in latest Qt though.

I was always afraid that it was hard to control the rendering. For D3D11 textures we explicitly set NV12 pixel formats for frames: https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavvideoframe.cpp#L421 And return 2 textures qavhwdevice_d3d11

So wondering if this is still relevant.

[geminixdev]

To make sure that I understand correctly:

1) you do not see this error / crash with #453 added in the video widget example (or elsewhere)? 2) And if not, what GPU? Intel / Nvidia / AMD?

And if you do not see it, does it seem that we now have full zero copy rendering there?

[geminixdev]

It seems to be fixed in latest Qt though.

I was always afraid that it was hard to control the rendering. For D3D11 textures we explicitly set NV12 pixel formats for frames: https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavvideoframe.cpp#L421 And return 2 textures qavhwdevice_d3d11

So wondering if this is still relevant.

I can follow what you say, and understand the question, but cannot answer (yet). Possibly a deep check of the Qt code can show what they try to do. (Hoping they do it correctly by now in the latest code).

[valbok]

To make sure that I understand correctly:

1. you do not see this error / crash with Support only AV_PIX_FMT_D3D11 for d3d11 #453 added in the video widget example (or elsewhere)?

I don't see any crashes

2. And if not, what GPU? Intel / Nvidia / AMD?

Nvidia

And if you do not see it, does it seem that we now have full zero copy rendering there?

For QML it was true even for 5.15 when RHI was introduced. But for widgets, I would need to check the code to confirm.

[valbok]

. (Hoping they do it correctly by now in the latest code).

We could try to avoid rendering, and see if it crashes anyway. F.e. keeping

QVideoFrame v = frame;
qDebug() << frame.handleType() << frame.handle();

and confirm that it does not crash.

[geminixdev]

. (Hoping they do it correctly by now in the latest code).

We could try to avoid rendering, and see if it crashes anyway. F.e. keeping

QVideoFrame v = frame;
qDebug() << frame.handleType() << frame.handle();

and confirm that it does not crash.

I think I understand, and will test this.

[geminixdev]

6. Btw I use the latest from https://github.com/BtbN/FFmpeg-Builds/releases

Yes, same. It wasn't the very latest, but I'll update to latest from today and retest.

I used the BtbN ffmpeg from December 2023, and updated now to the download from today.

And now the crash does not happen in the video widget example. In my app it is similar, not crashing, at least not yet in the short tests, but I can still provoke the DXGI_ERROR_DEVICE_HUNG device removal when I switch output windows. Not every time, but often.

However now there are semi transparent color stripe overlays in the video output, in the video widget example and in my App, usually violet, but occasionally whitish. Sometimes ot looks like the complete picture is whitish. See attachements.

In the Qt bugreports it was mentioned that the ffmpeg d3d11 decoding is using a texture array, and that they have difficulties to avoid that the same texture is read and written to. Looking at the pictures, this might be the issue.

My guess is that a deep look into the very latest Qt source and how they fixed this (if they did indeed go zero copy) might be required to see if anything can be done about this.

[geminixdev]

. (Hoping they do it correctly by now in the latest code).

We could try to avoid rendering, and see if it crashes anyway. F.e. keeping

QVideoFrame v = frame;
qDebug() << frame.handleType() << frame.handle();

and confirm that it does not crash.

I think I understand, and will test this.

I did not test this anymore, because with the very latest ffmpag the crash did not happen yet, see comment above.

[geminixdev]

It seems to be fixed in latest Qt though.

I was always afraid that it was hard to control the rendering. For D3D11 textures we explicitly set NV12 pixel formats for frames: https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavvideoframe.cpp#L421 And return 2 textures qavhwdevice_d3d11

So wondering if this is still relevant.

Or could it be a format issue with the 2 NV12 textures which get returned? As you asked, if this would "still be relevant"?

[geminixdev]

Possibly there is a hint there, about the ffmpeg texture handling:

https://code.qt.io/cgit/qt/qtmultimedia.git/tree/src/plugins/multimedia/ffmpeg/qffmpeghwaccel_d3d11.cpp

[valbok]

https://github.com/valbok/QtAVPlayer/pull/456 @geminixdev could you please check it out?

[geminixdev]

456 @geminixdev could you please check it out?

will test asap

[geminixdev]

456 @geminixdev could you please check it out?

Wow, thanks, this is excellent! It works!!!!!!!

No crash, no error messages, no color stripes. The picture looks correct.

Testing it also on a weak old Celeron PC with a 4K monitor, it plays smooth, it feels like zero copy rendering.

I will do more testing in the next days, but it looks all good now.