Dolby Vision - run as single layer

jartas commented 2 years ago

Hi, after fix 188 I try run Dolby Vision movie, there is problem with colors. I read about this problem and I read that Infuse use fake DV and run only HDR layer 10bit. Could you run only single HDR layer for DV video ?? Thank you

kingslay commented 2 years ago

Can you provide me this demo video. i try

jartas commented 2 years ago

Hi, i download sample from dolby, there is 4K Dolby Vision, problem is that color is pink and incorrectly detects dolby vision.

https://iscc.site/DVProf5_UHD_Dolby_Atmos.mp4

kingslay commented 2 years ago

This video is missing some parameters for video color. So there is something wrong with the displayed color. I also have this problem with vlc, but there is no problem with Apple's quicktime player. I'm sorry that I haven't found a solution for this problem. I will continue to try

kingslay commented 2 years ago

Duplicate of #150

Alanko5 commented 2 years ago

@kingslay

Please look at that:

Dolby Vision is implemented in FFmpeg+mpv (IPTPQc2 colorspace plus MMR reshaping, output to PQ surface works, reference with LG C9, that is reference v4 Dolby Vision implementation from Dolby), no support for (F)EL or dynamic metadata.

Do you think this is the solution for us? https://trac.ffmpeg.org/ticket/7624

HDR displays: mpv.com --target-trc=pq --target-prim=bt.2020 --vo=gpu-next --gpu-context=d3d11 file.mp4/mkv/ts

SDR gamma 2.2 display: mpv.com --target-trc=gamma2.2 --target-prim=bt.709 --vo=gpu-next --gpu-context=d3d11 file.mp4/mkv/ts

avsforum :

Since 2 days ago profile 5 and other even HLG profile 8 MMR decode to PQ in d3d11 context in mpv. Reference with LG C9. Congrats to me, haasn and others involved, like quitvoid for patches to ffmpeg.

mpv.com --target-trc=pq --target-prim=bt.2020 --vo=gpu-next --gpu-context=d3d11 file.mp4/mkv/ts

IPTPQc2

Alanko5 commented 2 years ago

https://code.videolan.org/videolan/libplacebo/-/issues/197

https://code.videolan.org/videolan/vlc/-/issues/26584

Alanko5 commented 2 years ago

@kingslay check this: https://github.com/quietvoid/vs-nlq

Here the mixing of BL and EL layer is done. The result is a 12-bit DV image.

Alanko5 commented 2 years ago

In your FFMPEG build not have -> dovi_rpu.c libavcodec/dovi_rpu.c

Inital commit for DolbiVision in FFMPEG https://github.com/FFmpeg/FFmpeg/commit/fe0403373964f6a5e633eb7adf9aa104cf9d59ff

lavc: Implement Dolby Vision RPU parsing Based on a mixture of guesswork, partial documentation in patents, and reverse engineering of real-world samples. Confirmed working for all the samples I've thrown at it.

Contains some annoying machinery to persist these values in between frames, which is needed in theory even though I've never actually seen a sample that relies on it in practice. May or may not work.

Since the distinction matters greatly for parsing the color matrix values, this includes a small helper function to guess the right profile from the RPU itself in case the user has forgotten to forward the dovi configuration record to the decoder. (Which in practice, only ffmpeg.c and ffplay do..)

Notable omissions / deviations:

CRC32 verification. This is based on the MPEG2 CRC32 type, which is similar to IEEE CRC32 but apparently different in subtle enough ways that I could not get it to pass verification no matter what parameters I fed to av_crc. It's possible the code needs some changes.
Linear interpolation support. Nothing documents this (beyond its existence) and no samples use it, so impossible to implement.
All of the extension metadata blocks, but these contain values that seem largely congruent with ST2094, HDR10, or other existing forms of side data, so I will defer parsing/attaching them to a future commit.
The patent describes a mechanism for predicting coefficients from previous RPUs, but the bit for the flag whether to use the prediction deltas or signal entirely new coefficients does not seem to be present in actual RPUs, so we ignore this subsystem entirely.
In the patent's spec, the NLQ subsystem also loops over num_nlq_pivots, but even in the patent the number is hard-coded to one iteration rather than signalled. So we only store one set of coefs.

kingslay commented 2 years ago

i had upgrade ffmpeg to 5.1. it had dovi_rpu

Alanko5 commented 2 years ago

I downloaded the latest version from git and it's not there.

https://github.com/kingslay/KSPlayer/commit/dae3e8d57cace18c5b4185035672b63fb7b67ec6#diff-e4e468a875016df080685ba4caa2a240bbe532939c226b5b349e88f17e41f179

kingslay commented 2 years ago

you can run nm -m KSPlayer/Sources/Libavcodec.xcframework/ios-arm64_arm64e/Libavcodec.framework/Libavcodec and search dovi_rpu

Alanko5 commented 2 years ago

kingslay commented 2 years ago

commit 5da0776117225d5c46c4d180ba20139e7a7267fb

kingslay commented 2 years ago

not dovi_rpu.h . is dovi_rpu.o

kingslay commented 2 years ago

/KSPlayer/Sources/Libavcodec.xcframework/ios-arm64_arm64e/Libavcodec.framework/Libavcodec:dovi_rpu.o: (undefined) external _av_buffer_allocz (undefined) external _av_buffer_create (undefined) external _av_buffer_replace (undefined) external _av_buffer_unref (undefined) external _av_dovi_metadata_alloc (undefined) external _av_frame_new_side_data_from_buf (undefined) external _av_free (undefined) external _av_log (undefined) external _avpriv_request_sample ---------------- (LTO,CODE) external _ff_dovi_attach_side_data ---------------- (LTO,CODE) external _ff_dovi_ctx_flush ---------------- (LTO,CODE) external _ff_dovi_ctx_replace ---------------- (LTO,CODE) external _ff_dovi_ctx_unref ---------------- (LTO,CODE) external _ff_dovi_rpu_parse ---------------- (LTO,CODE) external _ff_dovi_update_cfg (undefined) external _ff_golomb_vlc_len (undefined) external _ff_ue_golomb_vlc_code ---------------- (LTO,CODE) non-external _get_se_coef

Alanko5 commented 2 years ago

Then I don't understand why it doesn't work, DV detection and correct matrix setting.

kingslay commented 2 years ago

If DV's detection is correct, will Dolby Vision be displayed on TV without problems? @Alanko5

Alanko5 commented 2 years ago

yes, we know it will happen, but it's the first step...

can you advise how to do it?

Alanko5 commented 2 years ago

If DV's detection is correct, will Dolby Vision be displayed on TV without problems? @Alanko5

No, if DV dual layer or DV profile 5, the colors are wrong. If DV profile 8 single layer, colors is OK.

HDR colors are correct.

Alanko5 commented 2 years ago

If DV profile 8 single layer, colors is OK.

Not if it has rehhsaping, polinomial or if it has MMR.

do you have any ideas?

Alanko5 commented 1 year ago

https://developer.apple.com/av-foundation/Incorporating-HDR-video-with-Dolby-Vision-into-your-apps.pdf

Reading HDR metadata, for renderer/reshaper.

if let sideData = avframe.pointee.side_data.pointee?.pointee {
                        if sideData.type == AV_FRAME_DATA_DOVI_RPU_BUFFER {
                            let rpuBuff = sideData.data.withMemoryRebound(to: [UInt8].self, capacity: 1) { $0 }
                            print("")
                        }
                        if sideData.type == AV_FRAME_DATA_DOVI_METADATA { // AVDOVIMetadata
                            let doviMeta = sideData.data.withMemoryRebound(to: AVDOVIMetadata.self, capacity: 1) { $0 }
                            let header = av_dovi_get_header(doviMeta)
                            let mapping = av_dovi_get_mapping(doviMeta)
                            let color = av_dovi_get_color(doviMeta)
                            print("")
                        }
                        if sideData.type == AV_FRAME_DATA_DYNAMIC_HDR_PLUS { // AVDynamicHDRPlus
                            let hdrPlus = sideData.data.withMemoryRebound(to: AVDynamicHDRPlus.self, capacity: 1) { $0 }.pointee
                            print("")
                        }
                        if sideData.type == AV_FRAME_DATA_DYNAMIC_HDR_VIVID { // AVDynamicHDRVivid
                            let hdrVivid = sideData.data.withMemoryRebound(to: AVDynamicHDRVivid.self, capacity: 1) { $0 }.pointee
                            print("")
                        }

 }

Alanko5 commented 1 year ago

VLC implent DOVI

https://code.videolan.org/videolan/libplacebo/-/merge_requests/207/diffs

https://code.videolan.org/videolan/libplacebo/-/merge_requests/208/diffs

https://code.videolan.org/videolan/libplacebo/-/issues/113

kingslay commented 10 months ago

我现在接入了libplacebo。能否提供一个测试url。看下还会不会有问题

cdguy commented 10 months ago

我现在接入了libplacebo。能否提供一个测试url。看下还会不会有问题

I will try to find a video link for you @kingslay

cdguy commented 10 months ago

@kingslay I am downloading movies with different DV profiles. I want to be sure before I post something. I will wait until Monday and let you know about the results

cdguy commented 10 months ago

Hello @kingslay , with the help of @JeremTM93 , he tried quite a few movies and series on AppleTV and we found a correlation when the colours are greenish and purplish.

It's always happening when the Dolby Vision is “profile 5” (more precisely dvhe.05.06 BL+RPU) with bt.1886 gamma encoded in bt.601-525 regardless of the level (limited or full). // you can refer to this post too https://github.com/kingslay/KSPlayer/issues/348 //

When you open the links (provided via e-mail) via Infuse, the app creates a “fake DV output” and the colours looks normal. The TV info says that the videos are DV.

Please only make changes to “DV Profile 5” because other profiles work normally :)

cdguy commented 10 months ago

Ksplayer

Infuse (the TV says the content is DV, but we all know it is a fake DV)

kingslay commented 10 months ago

好的。谢谢你们的测试。我试着解决下profile 5的问题。

cdguy commented 10 months ago

It's always happening when the Dolby Vision is “profile 5” (more precisely dvhe.05.06 BL+RPU) with bt.1886 gamma encoded in bt.601-525 regardless of the level (limited or full). // you can refer to this post too #348 //

Here is a prompt I found; maybe it would be helpful:

To correct the greenish and purplish colors of a video encoded in dvhe.05.06 BL+RPU with bt.1886 gamma encoded in bt.601-525 to bt.2020 gamma using ffmpeg with libplacebo without losing the Dolby Vision properties, you can use the following command:

ffmpeg -i input.mkv -vf "zscale=matrixin=bt470bg:matrix=bt2020,format=gbrpf32le,zscale=matrixin=bt2020:matrix=bt2020nc,format=yuv420p" -c:v libplacebo -pix_fmt yuv420p -colorspace bt2020 -color_primaries bt2020 -color_trc bt2020 -color_range tv -c:a copy output.mkv

This command uses the zscale filter to convert the colorspace and gamma of the input video to bt.2020, and then sets the output colorspace, primaries, transfer characteristics, and color range to bt.2020. The -c:a copy option is used to copy the audio stream without re-encoding, preserving the Dolby Vision properties of the video

kingslay commented 10 months ago

用libplacebo这个filter，是可以播放profile 5。但是目前我试了只能在macOS上才可以。在iOS/tvOS都会crash。

karelrooted commented 10 months ago

用libplacebo这个filter，是可以播放profile 5。但是目前我试了只能在macOS上才可以。在iOS/tvOS都会crash。

iOS is a bit weird，crash under xcode debug mode，but if you open app normaly without xcode debug，it works fine（tvos crash with or without debug），i fire a issue on libplacebo repo（ https://github.com/haasn/libplacebo/issues/225 ）， if you are familar with vulkan， may be you can try if vvl has some info

kingslay commented 10 months ago

@karelrooted 我试了下，确实在iOS，只要不是debug。是不会crash的。谢谢你提供的信息。另外你提供的关于libshaderc和libsmbclient的编译补丁很棒。解决了我的编译问题。

karelrooted commented 10 months ago

@karelrooted 我试了下，确实在iOS，只要不是debug。是不会crash的。谢谢你提供的信息。另外你提供的关于libshaderc和libsmbclient的编译补丁很棒。解决了我的编译问题。

samba is still a WIP, have to figure out how to build a combined static lib or dynamic lib, xbmc use a old version of samba, or we can just borrow build process from xbmc (if user enable samba , the license become GPL v3 anyway ,so borrow build script from xbmc is not a problem)

cdguy commented 10 months ago

@kingslay with the last commits, the colors are correctly (on IOS) displayed, but I guess you did not close this thread because there are dropped frames during playback, right?

karelrooted commented 10 months ago

@kingslay with the last commits, the colors are correctly (on IOS) displayed, but I guess you did not close this thread because there are dropped frames during playback, right?

once started to copy the frame from hardware to system memory, the performance started to degrade draftily, tested it on Apple TV 4K 2017, iPad Air 4, MacBook Pro 2018 15inch , no one device can play Dolby vision profile 5 video smoothly with copy back to system memory, already try "-hwaccel=videotoolbox hwupload=derive_device=vulkan" , did not work , have to copy back to system memory first

mpv's vo=gpu-next seems works smoothly, if old device's a concern, may be we need to use libplacebo's vulkan render to render the decoded frame from videotoolbox instead of use ffmpeg's libplacebo filter and then render it on MetalLayer

kingslay commented 10 months ago

主要是因为tvOS还不能用这个功能。所以我没有把它关了

debugly commented 9 months ago

用libplacebo这个filter，是可以播放profile 5。但是目前我试了只能在macOS上才可以。在iOS/tvOS都会crash。

我运行了macOS端的demo，请问如何开启 placebo filter呢，我这有个 profile5的视频，直接运行demo渲染出来的颜色不对。

kingslay commented 9 months ago

要自己加filter libplacebo=color_primaries=9:color_trc=14

karelrooted commented 9 months ago

please note tvOS crash bug in libplacebo filter is fixed with motlenvk latest git

the Xcode debug mode crash bug is not fixed though, still can't get the macOS version running against official Vulkan SDK to work, so can't get the related vvl info, if any of you manager to get it running and find some info in the vvl , please post it in the issue

kingslay commented 9 months ago

谢谢你的告知。我升级下motlenvk 看下行不行。

cdguy commented 9 months ago

when you are done with all updates, can you please update the testflight app; @JeremTM93 is the best person that can run all DV profile tests and let you know if something is wrong

karelrooted commented 9 months ago

@kingslay is performance a issue on Apple TV 4K 2022 ? on Apple TV 4K 2017 , is basically unusable

I also test it on mpv with vo=videotoolbox-copy & vf=libplacebo, is basically unusable too with vo=gpu-next , with default profile , there is a lot of frame drop , after change to fast profile, the video play fines though

cdguy commented 9 months ago

@kingslay is performance a issue on Apple TV 4K 2022 ? on Apple TV 4K 2017 , is basically unusable

I also test it on mpv with vo=videotoolbox-copy & vf=libplacebo, is basically unusable too with vo=gpu-next , with default profile , there is a lot of frame drop , after change to fast profile, the video play fines though

I was about to write something similar:

on ios device the colour output is very good however almost every 7-8 seconds I have almost 100 frame drops

kingslay commented 9 months ago

@kingslay is performance a issue on Apple TV 4K 2022 ? on Apple TV 4K 2017 , is basically unusable

I also test it on mpv with vo=videotoolbox-copy & vf=libplacebo, is basically unusable too with vo=gpu-next , with default profile , there is a lot of frame drop , after change to fast profile, the video play fines though

@karelrooted 你的意思是，你对mpv 配置了vo=gpu-next & --profile=sw-fast 之后。视频播放就很流畅了，是不是。

karelrooted commented 9 months ago

@karelrooted 你的意思是，你对mpv 配置了vo=gpu-next & --profile=sw-fast 之后。视频播放就很流畅了，是不是。

yes，with --profile=fast,video play smoothly on Apple TV 4K 2017

If performance is a issue, may be we need to render the decoded frame from videotoolbox directly with libplacebo render(like mpv's gpunext) (ffplay's latest git also has libplacebo render, I did't check if it has videotoolbox and libplacebo interlop)

kingslay commented 9 months ago

目前我用的画面渲染有两个 AVSampleBufferDisplayLayer和CAMetalLayer。所以要解决这个性能问题的话，那有两个技术方案：

增加libplacebo 渲染器。libplacebo渲染器应该是在渲染的时候顺便做了颜色的转化，这样才避免了gpu和cup的内存切换。
只针对profile 5这个格式，把libplacebo=color_primaries=9:color_trc=14的颜色转化，用metal来实现一下。继续使用CAMetalLayer来进行渲染。

karelrooted commented 9 months ago

目前我用的画面渲染有两个 AVSampleBufferDisplayLayer和CAMetalLayer。所以要解决这个性能问题的话，那有两个技术方案：

增加libplacebo 渲染器。libplacebo渲染器应该是在渲染的时候顺便做了颜色的转化，这样才避免了gpu和cup的内存切换。

只针对profile 5这个格式，把libplacebo=color_primaries=9:color_trc=14的颜色转化，用metal来实现一下。继续使用CAMetalLayer来进行渲染。

or we can wait for ffmpeg to fix videotoolbox and libplacebo interlop， if "-hwaccel=videotoolbox -hwupload=derive_device=vulkan" works，then performance issue is gone

the libplacebo part that make prpfile 5 color correct is vf=libplacebo=apply_dolbyvision=1， which is enable by default, so just use vf=libplacebo is enough for our purpose

did you test it on Apple TV 4K 2022? does it work on new more powerful device?

cdguy commented 9 months ago

目前我用的画面渲染有两个 AVSampleBufferDisplayLayer和CAMetalLayer。所以要解决这个性能问题的话，那有两个技术方案：

增加libplacebo 渲染器。libplacebo渲染器应该是在渲染的时候顺便做了颜色的转化，这样才避免了gpu和cup的内存切换。

只针对profile 5这个格式，把libplacebo=color_primaries=9:color_trc=14的颜色转化，用metal来实现一下。继续使用CAMetalLayer来进行渲染。

or we can wait for ffmpeg to fix videotoolbox and libplacebo interlop， if "-hwaccel=videotoolbox -hwupload=derive_device=vulkan" works，then performance issue is gone

the libplacebo part that make prpfile 5 color correct is vf=libplacebo=apply_dolbyvision=1， which is enable by default, so just use vf=libplacebo is enough for our purpose

did you test it on Apple TV 4K 2022? does it work on new more powerful device?

@kingslay asking just in case; have you tried the code suggested above? ("-hwaccel=videotoolbox -hwupload=derive_device=vulkan")

kingslay commented 9 months ago

有的，但是这样是无法运行的。可能要等待ffmpeg支持了才可以

kingslay commented 8 months ago

目前ffmpeg 6.1.1还是不支持把vulkan当作解码器。我看ffmpeg的最新代码，是有开始支持用vulkan来进行渲染了。可以在等下一个版本

kingslay / KSPlayer

Dolby Vision - run as single layer #190