FFmpegInteropX plays some videos with abnormal colors or black screen

[yikuo123]

I encountered a problem when using FFmpegInteropX to play some videos.

If I choose Automatic or ForceFFmpegSoftwareDecoder, the video color is abnormal and there is a mosaic phenomenon;

if I choose AutomaticSystemDecoder or ForceSystemDecoder, it will be black screen.

But if I use the system native player or ffplay to play, everything is normal.

The video information obtained by ffprobe is as follows:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\Video.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: isommp41mp42
    creation_time   : 2022-08-06T01:13:14.000000Z
    copyright       :
    copyright-eng   :
  Duration: 00:02:37.03, start: 0.000000, bitrate: 9796 kb/s
  Stream #0:0[0x1](und): Video: hevc (Main 10) (hvc1 / 0x31637668), yuv420p10le(tv, bt2020nc/bt2020/arib-std-b67), 1920x1072, 9669 kb/s, 28.99 fps, 29 tbr, 600 tbn (default)
    Metadata:
      creation_time   : 2022-08-06T01:13:14.000000Z
      handler_name    : Core Media Video
      vendor_id       : [0][0][0][0]
    Side data:
      DOVI configuration record: version: 1.0, profile: 8, level: 4, rpu flag: 1, el flag: 0, bl flag: 1, compatibility id: 4
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 120 kb/s (default)
    Metadata:
      creation_time   : 2022-08-06T01:13:14.000000Z
      handler_name    : Core Media Audio
      vendor_id       : [0][0][0][0]

[brabebhin]

Please upload a sample video somewhere.

[yikuo123]

@brabebhin Hello, this is the sample video:

https://tmp.link/f/6409a4e207e50

[brabebhin]

Thanks. I downloaded the video, but i won't be able to take a look until Monday.

Of the top of my head, here's some possible causes :

The video had some issues during muxing, so any GPU based decoding will fail or have weird results. You can try remuxing it (convert to another format and then back to hevc) and see if that fixes it.

GPU drivers might be bugged.

Some hidden edge case in our lib or ffmpeg.

[yikuo123]

Thanks. I downloaded the video, but i won't be able to take a look until Monday.

Of the top of my head, here's some possible causes :

The video had some issues during muxing, so any GPU based decoding will fail or have weird results. You can try remuxing it (convert to another fornat and then back to hevc) and see if that fixes it.

GPU drivers might be bugged.

Some hidden edge case in our lib or ffmpeg.

Thanks for your reply. I think this problem may not have much to do with GPU-based decoding, because ffmpeg software decoding is also not very normal. And I also think that this type of video may have some differences from most videos in some places, which just triggered some unhandled problems when ffmpeg decodes.

[brabebhin]

Ffmpeg software decoding can fix issues in the file up to some point, which is not possible for hardware. The fact that system pass through also doesn't work points to faulty file. You can trim the last second to conserve the file but pass it through a remux and see what happens.

[lukasf]

I do not get any kind of artifacts with the sample file, no matter if I use hardware or software decoding. Have you checked that the sample still has those issues @yikuo123? If you cut samples using ffmpeg, it will do a remux the stream. This can often resolve issues with incorrectly muxed files, so some problems cannot be reproduced when creating samples with ffmpeg.

If the sample plays fine you your side as well, try to do a binary split of the file and see if the first chunk shows the issue. Often this works. Or upload the complete file, if it is possible. If you have WSL, it's very easy to split, e.g. split -b 100MB video.mov. Windows does not have any built in tools afaik.

[yikuo123]

@lukasf

Yes, I still encounter color anomalies (Automatic/ForceFFmpegSoftwareDecoder) or black screen (AutomaticSystemDecoder/ForceSystemDecoder) when playing this video file.

I have tried it on both my PC and Xbox, and the result is the same.

I have also tried splitting the file into smaller chunks using the split command, but playing the first chunk still shows the same phenomenon.

[lukasf]

Okay, looks like this is a HDR video and it currently does not play well when the display does not support HDR (or HDR is not enabled). Actually, on my PC this video looks a lot better when playing with our lib, compared to Windows 11 media player or MPC/VLC, because all other players do not use the HDR color info.

However, I need to disable forwarding the HDR metdatata when the display does not have HDR enabled. I have made a PR #339 for this. Please check if this works on your side as well. For me, the video now plays well, both with HDR enabled and disabled (but it looks a lot better with HDR enabled, of course).

[yikuo123]

Okay, looks like this is a HDR video and it currently does not play well when the display does not support HDR (or HDR is not enabled). Actually, on my PC this video looks a lot better when playing with our lib, compared to Windows 11 media player or MPC/VLC, because all other players do not use the HDR color info.

However, I need to disable forwarding the HDR metdatata when the display does not have HDR enabled. I have made a PR #339 for this. Please check if this works on your side as well. For me, the video now plays well, both with HDR enabled and disabled (but it looks a lot better with HDR enabled, of course).

Thank you for your PR, I have pulled and verified it, and now the color abnormality problem is solved. Both Automatic and ForceFFmpegSoftwareDecoder modes can play normally.

But AutomaticSystemDecoder and ForceSystemDecoder still show black screen, whether I have installed HEVCVideoExtension (Microsoft.HEVCVideoExtension_8wekyb3d8bbwe) or not. If I use MediaSource instead of FFmpegMediaSource, it works well.

[brabebhin]

Unfortunately i could reproduce the problem, but since this is a pass through, we can't really know what's wrong. I don't see any errors in the logs or outputs. I will dig some more.

[lukasf]

Closing this. System decoders is legacy and not recommended anymore and playback works fine with ffmpeg hw decoding.