FFmpeg decoding leads to invalid data when processing input with D3D11VA and DXVA2 hw acceleration

[streamingdv]

The last 2 weeks I was trying to figure out why I can't decode some h264 keyframes when using D3D11VA or DXVA2 hw acceleration and I'm still not sure if I'm on the right track but I guess I found the potential issue but not sure.

In my project I use JavaCV 1.5.8 to decode an h264 stream and everything works fine when using sw decoding or cuvid decoding but as soon as I use D3D11VA or DXVA2 hw acceleration some key frames can't be decoded anymore because I will receive an "invalid data found when processing input" when calling "_avcodec_sendpacket". I have build a minimal reproducable example which can be downloaed from here. Here is a link to just the h264 keyframe with sps and pps prepended.

To run the project just perform a .\gradlew build and afterwards a .\gradlew run. The test project is trying to decode one h264 keyframe (with sps and pps). If you run the example the last log message shown in the terminal should be "SUCESS with HW decoding". The hardware decoder can be changed via the HW_DEVICE_TYPE variable in the App.java class. To disable hw acceleration just set the USE_HW_ACCEL to false. You can force the decoder to spit out the frame by changing USE_AV_EF_EXPLODE to false.

After countless of hours researching on the internet I think I finally found out what's the issue. On another project here on github I found this issue

https://github.com/wang-bin/QtAV/issues/923

which is exactly the problem I'm experiencing as well. It seems that in h264dec.h there is a variable called MAX_SLICES which is set to 32. This MAX_SLICES value is used here dxva2_h264.c. I also found this interesting entry which seems to be related to that issue

https://trac.ffmpeg.org/ticket/9771#no1

I'm not an expert in ffmpeg so my assumptations can be wrong but I think my issue is related to the problems described above. Would it be possible to increase the MAX_SLICES to something higher like 128 or 256 with a patch in general? Or are there any other possibilities? As I get this h264 stream frorm 3rd Party I can't change the format but I don't think it will be possible to use D3D11VA or DXVA2 otherwise with this small max slices support.

[saudet]

Sounds like a small enough change. Please do open a pull request with a patch, or actually a call to sedinplace in cppbuild.sh is probably easiest.

[streamingdv]

@saudet okay sounds good. I made myself a bit familiar with the cppbuild script and as far as I have understood everything correctly this line should do the trick

sedinplace 's/#define MAX_SLICES 32/#define MAX_SLICES 256/g' ffmpeg-*/libavcodec/h264dec.h

Where exactly should I place this line? Below this line maybe? Are there any preferences?

[saudet]

Yes, that's perfect!

[streamingdv]

@saudet small pull request available

https://github.com/bytedeco/javacpp-presets/pull/1349

[streamingdv]

Is there anything that needs to be done on my side in order to get my small pull request merged? I'm not in a hurry and I totally understand that there are other priorities or maybe fixed times when things are merged. Just want to make sure I did not forget anything :)

[saudet]

Nope, it's fine! Thanks. I've just merged, so please give it a try with the snapshots: http://bytedeco.org/builds/

[streamingdv]

Problem solved :)