HDR videos format SMPTE ST 2086 looks "inverted" or in "negative"

[MrDiamond10]

The bug

Dear developers,

I have a bug in case of viewing HDR videos in format SMPTE ST 2086.

I made a little investigation about problem between few videos. I checked metadata for a differences and I always end up with different HDR format. Other format then SMPTE ST 2086 works fine. I think that transcoding of videos could help but when I tried it, it does not change, even videos were unplayable in Immich. I always tested source file which was everytime OK. In screenshot you can see what I mean when I say "inverted" or in "negative".

Issue is always present in PC web browser. But in app in mobile (Android 14), it is only "half" issue, because in gallery thumbnails looks bad, but after opening the video, it looks fine. (screenshot attached)

I checked more videos and compared its metadata and it looks that issue is only that HDR format.

What do you recommend please?

---

Metadata of problematic reference video: General Complete name : G:\Server\Immich Github issue\20230820_173401.mp4 Format : MPEG-4 Format profile : Base Media / Version 2 Codec ID : mp42 (isom/mp42) File size : 15.5 MiB Duration : 7 s 342 ms Overall bit rate : 17.7 Mb/s Frame rate : 23.976 FPS Encoded date : 2023-08-20 15:34:10 UTC Tagged date : 2023-08-20 15:34:10 UTC xyz : +48.1993+017.1907/ com.android.version : 13

Video ID : 2 Format : HEVC Format/Info : High Efficiency Video Coding Format profile : Main 10@L1@Main HDR format : SMPTE ST 2086 Codec ID : hvc1 Codec ID/Info : High Efficiency Video Coding Duration : 7 s 342 ms Bit rate : 17.5 Mb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Rotation : 90° Frame rate mode : Variable Frame rate : 23.976 (23976/1000) FPS Minimum frame rate : 23.911 FPS Maximum frame rate : 24.038 FPS Standard : NTSC Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 10 bits Bits/(Pixel*Frame) : 0.353 Stream size : 15.3 MiB (99%) Title : VideoHandle Language : English Encoded date : 2023-08-20 15:34:10 UTC Tagged date : 2023-08-20 15:34:10 UTC Color range : Limited Color primaries : BT.2020 Transfer characteristics : HLG Matrix coefficients : BT.2020 non-constant Mastering display color primaries : R: x=0.000020 y=0.000020, G: x=0.000020 y=0.000020, B: x=0.000020 y=0.000020, White point: x=0.000020 y=0.000020 Mastering display luminance : min: 0.0001 cd/m2, max: 1 cd/m2 mdhd_Duration : 7342 Codec configuration box : hvcC

Audio ID : 1 Format : AAC LC Format/Info : Advanced Audio Codec Low Complexity Codec ID : mp4a-40-2 Duration : 7 s 122 ms Source duration : 7 s 104 ms Bit rate mode : Constant Bit rate : 156 kb/s Channel(s) : 2 channels Channel layout : L R Sampling rate : 48.0 kHz Frame rate : 46.875 FPS (1024 SPF) Compression mode : Lossy Stream size : 136 KiB (1%) Source stream size : 136 KiB (1%) Title : SoundHandle Language : English Encoded date : 2023-08-20 15:34:10 UTC Tagged date : 2023-08-20 15:34:10 UTC mdhd_Duration : 7122

---

Metadata of similar video, which does not have that issue:

General Complete name : G:\Server\Immich Github issue\good2 20230212_022238.mp4 Format : MPEG-4 Format profile : Base Media / Version 2 Codec ID : mp42 (isom/mp42) File size : 43.5 MiB Duration : 12 s 82 ms Overall bit rate : 30.2 Mb/s Frame rate : 59.940 FPS Encoded date : 2023-02-12 01:22:51 UTC Tagged date : 2023-02-12 01:22:51 UTC xyz : +48.7296+019.1353/ com.android.version : 13

Video ID : 2 Format : AVC Format/Info : Advanced Video Codec Format profile : High@L1 Format settings : CABAC / 1 Ref Frames Format settings, CABAC : Yes Format settings, Reference frames : 1 frame Format settings, GOP : M=1, N=60 Codec ID : avc1 Codec ID/Info : Advanced Video Coding Duration : 12 s 82 ms Bit rate : 30.1 Mb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Rotation : 90° Frame rate mode : Variable Frame rate : 59.940 (59940/1000) FPS Minimum frame rate : 59.563 FPS Maximum frame rate : 60.281 FPS Standard : NTSC Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 8 bits Scan type : Progressive Bits/(Pixel*Frame) : 0.242 Stream size : 43.3 MiB (99%) Title : VideoHandle Language : English Encoded date : 2023-02-12 01:22:51 UTC Tagged date : 2023-02-12 01:22:51 UTC Color range : Limited Color primaries : BT.709 Transfer characteristics : BT.709 Matrix coefficients : BT.709 mdhd_Duration : 12081 Codec configuration box : avcC

Audio ID : 1 Format : AAC LC Format/Info : Advanced Audio Codec Low Complexity Codec ID : mp4a-40-2 Duration : 12 s 78 ms Source duration : 12 s 75 ms Bit rate mode : Constant Bit rate : 156 kb/s Channel(s) : 2 channels Channel layout : L R Sampling rate : 48.0 kHz Frame rate : 46.875 FPS (1024 SPF) Compression mode : Lossy Stream size : 230 KiB (0%) Source stream size : 230 KiB (0%) Title : SoundHandle Language : English Encoded date : 2023-02-12 01:22:51 UTC Tagged date : 2023-02-12 01:22:51 UTC mdhd_Duration : 12078

---

Screenshot from web browser:

Video from web browser:

https://github.com/immich-app/immich/assets/164943054/124afb10-202e-48c0-afea-54be30f81254

---

Screenshot from APP:

---

Video from APP:

https://github.com/immich-app/immich/assets/164943054/bdbbe424-6203-4b50-a4a5-45742f671c4a

The OS that Immich Server is running on

TrueNAS-SCALE-24.04-RC.1

Version of Immich Server

v1.100.0

Version of Immich Mobile App

v1.100.0

Platform with the issue

• [X] Server
• [X] Web
• [X] Mobile

Your docker-compose.yml content

Default

Your .env content

Default

Reproduction steps

1.Upload video with SMPTE ST 2086 HDR format
2.check video in web browser and app

Additional information

Transcoding video to HABLE, which should support SMPTE ST 2086 HD format does not help. To REINHARD as well.

Issue was present even on 2 older versions of Immich

[mertalev]

This is very strange. Are you using any non-default transcoding settings? In particular are you using hardware acceleration? Also, could you share a sample video I can use to reproduce this locally?

[MrDiamond10]

This is very strange. Are you using any non-default transcoding settings? In particular are you using hardware acceleration? Also, could you share a sample video I can use to reproduce this locally?

Here you can download sample video: https://n.cloudklik.top/s/Ss7EbgbrW2YLQr4

It started with default transcoding settings, then I tried to solve it by changing few of them. Actual settings are on 4 attached screenshots. But in overal I changed that all videos should be transcoded and PRESET (-preset) from ultrafast to fast.

Hardware acceleration is not used as I undrestand settings correctly.

[mertalev]

I did some testing with this and found that the issue happens when tone-mapping these videos. It can't interpret the peak brightness of the video correctly and thinks it's 0.005 nits (impossibly dark). I'm not sure if this is bad metadata or if FFmpeg has a bug here. Interestingly, the mobius algorithm is able to handle this somewhat better than hable or reinhard, but the closest result is just with tone-mapping disabled.

I would disable tone-mapping and re-run both transcoding and thumbnail generation for these assets. If you don't expect more videos of this format, you can turn it back on after you're done so other HDR videos continue to get tone-mapped.

[mertalev]

Update: I tested with a bleeding edge FFmpeg build and it tone-mapped correctly. It looks like it was an FFmpeg bug after all.

[MrDiamond10]

Update: I tested with a bleeding edge FFmpeg build and it tone-mapped correctly. It looks like it was an FFmpeg bug after all.

Thank you very much. I have a question, maybe stupid one. How can I set or run(?) that bleeding edge FFmpeg build?

[mertalev]

You would have to make a Dockerfile extending from the microservices image that installs the bleeding edge FFmpeg version. I used the master variant from this page. You would also need to update the PATH so invoking ffmpeg calls this version instead of the default build in immich.

[MrDiamond10]

You would have to make a Dockerfile extending from the microservices image that installs the bleeding edge FFmpeg version. I used the master variant from this page. You would also need to update the PATH so invoking ffmpeg calls this version instead of the default build in immich.

Thank you for your help. Can you please describe it better? I an new in TrueNAS and Kubernetes, so little help would be great.

[mertalev]

Keep in mind that this requires you to use a custom chart and will hence be more complicated than your existing setup. I don't recommend it if you're new to all of this and want something simple.

[mertalev]

This should be fixed with #9402. It uses a different library for tone-mapping that doesn't have this issue.

[mertalev]

Sorry, we ended up not using that library. We might use it for a different hardware API in the future, or upgrade to 7.0 once Jellyfin makes that upgrade.