Closed FCLC closed 1 year ago
ffmpeg config (latest available from upstream as of 2022/04/16):
ffmpeg version N-106610-g24144af2dd Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 12.0.1 (GCC) 20220412 (experimental)
configuration: --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-ladspa --enable-libaom --enable-libass --enable-libcodec2 --enable-libdav1d --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-sdl2 --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-libx264 --enable-opencl --cc=gcc-12 --enable-libmfx --extra-cflags=-Wno-unused-function --enable-nonfree
source file:https://4kmedia.org/sony-swordsmith-hdr-uhd-4k-demo/
Will look into it and update it.
Update to this:
seems it also effects h264, mpeg2, mjpeg as well as h264_qsv.
haven't had time to test more, will follow up time permitting
tonemap_vaapi=format=nv12:primaries=bt2020:t=bt709
What are you expecting to achieve with those parameters?
Thanks for the info. Will check this issue and provide feedback this week.
tonemap_vaapi=format=nv12:primaries=bt2020:t=bt709
What are you expecting to achieve with those parameters?
1. Rec. 2020 defines a bit depth of either 10 or 12 bits => but nv12 is an 8bit format - it doesn't make sense to use 2020 primaries with that pixel format 2. Using bt709 transfer characteristics in combination with 2020 primaries => I don't think that this is valid at all and even if it was, it's still pretty unusual.
The above example is one of many. The same happens when using:
ffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'tonemap_vaapi=format=p010:primaries=bt2020,scale_vaapi=format=p010' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
(note that every part of the filter has explicit setting of 10bit colour)
The reason that the initial example was set the way it was was for control of variables. Per documentation, the tonemap_vaapi
filter defaults to outputting in nv12 when nothing is explicitly set.
See https://ffmpeg.org/ffmpeg-filters.html#tonemap_005fvaapi
The filter has 4 parameters: Format, Primaries, Transfer and Matrix.
Of those 4, Format defaults to nv12
, Primaries defaults to same as input
, Transfer defaults to bt709
and Matrix defaults to same as input
.
What this means is that, for most (all?) cases where HDR10 content is passed to the filter, the filter will be initialized with:
...PREVIOUS_FILTER,tonemap_vaapi=format=nv12:primaries=bt2020:transfer=bt709:matrix=bt2020nc,NEXT_FILTER...
When we then pass this to a 10bit capable encoder in a 10 bit capable format (say we use p010,bt2020,bt2020,bt2020nc
and pass that to hevc_vaapi) we still have the same issue as above, the resulting command and output being:
ffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'tonemap_vaapi=format=p010:primaries=bt2020:transfer=bt2020-10:matrix=bt2020nc' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
The above, with the singular change of primaries to bt709
with command ffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'tonemap_vaapi=format=p010:primaries=bt709:transfer=bt2020-10:matrix=bt2020nc' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
Which then results in:
The above example is one of many.
There are not really that many examples. You cannot arbitrarily combine all those parameters to get a valid result.
I think the primaries parameter is what switches tonemap_vaapi between HDR-to-HDR and HDR-to-SDR tone mapping.
The same happens when using:
ffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'tonemap_vaapi=format=p010:primaries=bt2020,scale_vaapi=format=p010' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
(note that every part of the filter has explicit setting of 10bit colour)
For encoding 10 bit HEVC, you might need to set the HevcProfileMain10 profile (I'm not sure whether it's auto-selected otherwise). If that doesn't fix it, then there might be in fact a bug. I'm always only using HDR-to-SDR tone mapping, so I have no experience with that case.
But some of your command lines just don't make sense, I would really focus on that one (I quoted) if HDR-to-HDR is your goal.
@FCLC , from the description, You cannot arbitrarily combine all those parameters to get a valid result. Need to set those values according the info.
Please see the description of VAAPI about the description of VAAPI: VAProcColorStandardType https://github.com/intel/libva/blob/master/va/va_vpp.h#L416 _VAProcColorProperties https://github.com/intel/libva/blob/master/va/va_vpp.h#L683 HDR10 needs to set the value as the below. And it should be 10bit or 12bit instead of 8bit. /* \brief Explicitly specified color properties.
@xhaihao , from @softworkz , HDR-to-SDR is working. Have you folks once validated HDR-To-HDR from FFMPEG side? Thanks.
@xhaihao , from @softworkz , HDR-to-SDR is working. Have you folks once validated HDR-To-HDR from FFMPEG side? Thanks.
I've never tried this (no use case for it).
@softworkz , you have never tried HDR to HDR, or HDR to SDR? Thanks in advance!
Always and only ever HDR-to-SDR.
@FCLC , https://ffmpeg.org/ffmpeg-filters.html#tonemap_005fvaapi
This example is correct. HDR(HDR10) video to bt2020-transfer-characteristic p010 format tonemap_vaapi=format=p010:t=bt2020-10
If HDR10 input, the value should be like this Format: 10bit or 12bit, P010 decoded raw data primaries, p: bt2020 transfer, t: ST2084.
Please use the correct combination, thanks in advance!
Always and only ever HDR-to-SDR.
Thanks @softworkz !
My alder lake chip is being sent out for RMA currently after dying the other morning; I wonder if that has/had something to do with the issues I was seing
@FCLC , I am curious what you are working on? Is it a company or individual? What kinds of product you are for? Thanks.
@FCLC , I am curious what you are working on? Is it a company or individual? What kinds of product you are for? Thanks.
I'm independent and working mostly on AVX-512FP16 accelerated SIMD support for libraries in HPC environments. I also happen to be a pretty big movie guy, so projects like FFMPEG have been a recent interest of mine. (Especially when it comes to maximizing usage of AVX2/AVX512 to better utilize these next gen ISA's.)
As for what I'm working on, I'm currently finishing a graduate program and hoping to be back into industry in either August or December.
I wonder what the use case would be for HDR-to-HDR tone mapping?
@FCLC , from the description, You cannot arbitrarily combine all those parameters to get a valid result. Need to set those values according the info.
Please see the description of VAAPI about the description of VAAPI: VAProcColorStandardType https://github.com/intel/libva/blob/master/va/va_vpp.h#L416 _VAProcColorProperties https://github.com/intel/libva/blob/master/va/va_vpp.h#L683 HDR10 needs to set the value as the below. And it should be 10bit or 12bit instead of 8bit. /* \brief Explicitly specified color properties. Use corresponding color properties section. For example, HDR10 content: colour_primaries = 9 (BT2020) transfer_characteristics = 16 (SMPTE ST2084) matrix_coefficients = 9 / VAProcColorStandardExplicit, And more, there is sample code about how to use HDR10 Tone Mapping: https://github.com/intel/libva-utils/blob/master/videoprocess/vpphdr_tm.cpp
Yup, one the same page!
What I'm getting at is that, when the tonemap_vaapi filter is used for HDR-SDR, we face a decision on how to handle the content.
if it's going to be sent to H264, VP8, mpeg2, mjpeg
then we known it's going to be nv12/8 bit
. (technically H264 has the option for 10-bit, but it's so rare as to be irrelevant for this discussion, as well as not supported in hardware.)
if it's going to be sent to HEVC/AV1/VP9
it can be 10-bit or 12-bit
if it is 10-bit, we use p010
if it is 12-bit we use [unknown,I think we use a 16-bit format. @softworkz do you know offhand? if not I'll ping Lynne] EDIT: heard back from the ffmpeg IRC, we think it's in p016
The filter by default is spec'd to return the base format of the input file, unless otherwise changed.
10bit, HDR10 source in HEVC container, tonemapped and sent to SDR 8bit hevc:
Decoded (either directly by a hardware decoder or by a software decoder then uploaded to GPU memory)
filter calls intel VA driver VA driver looks at the tonemapping information in the source file VA driver uses tonemapping information to apply a hard map to the input file
Input file is now 10bit SDR (p010), in other words a standard 10bit hardware stream in p010
if we pass to an hevc container, it implies 8,10 or 12 bit.
by default, output is mapped to 8 bit
We need to convert from 10bit to 8 bit, or in other words format p010 -> format nv12
However the command is equivalent to
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'tonemap_vaapi' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
The tonemap filter, when not given specific args, will default to the source files for some options, and imply it's own for others.
This is tonemap_vaapi=format=nv12:primaries=[SAME AS INPUT]:transfer=bt709:matrix=[SAME AS INPUT]
(Format defaults to nv12, Primaries defaults to same as input, Transfer defaults to bt709 and Matrix defaults to same as input.)
When I tested above, that resulted in:
ffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'tonemap_vaapi=format=p010:primaries=bt2020:transfer=bt2020-10:matrix=bt2020nc' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
and the output was
To me anyway, that looks like something isnt being propagated to the encoder OR the filter is having issues.
It's possible that the default format to nv12 is an issue.
Perhaps an auto insert scale filter needs to be added to pure hardware pipelines?
HEVC_vaapi has the following options which could help/be relevant:
-profile <int> E..V....... Set profile (general_profile_idc) (from -99 to 255) (default -99)
main 1 E..V.......
main10 2 E..V.......
rext
-tier <int> E..V....... Set tier (general_tier_flag) (from 0 to 1) (default main)
main 0 E..V.......
high 1 E..V.......
-sei <flags> E..V....... Set SEI to include (default hdr)
hdr E..V....... Include HDR metadata for mastering display colour volume and content light
I wonder what the use case would be for HDR-to-HDR tone mapping?
but I can see a scenario for conversion from HDR10+ to HDR10? Otherwise I can also see a use case for cointaner swapping, something like AV1_12 to HEVC_10?
Say we had a source that was HDR10+(not currently supported, but on roadmap I believe) in a 12bit AV1 container
if we want to keep the metadata, I think we need to still apply some of the tonemapping data to the source file before converting from HDR10+ 12 bit to HDR10
I wonder what the use case would be for HDR-to-HDR tone mapping?
but I can see a scenario for conversion from HDR10+ to HDR10? Otherwise I can also see a use case for cointaner swapping, something like AV1_12 to HEVC_10?
Say we had a source that was HDR10+(not currently supported, but on roadmap I believe) in a 12bit AV1 container
if we want to keep the metadata, I think we need to still apply some of the tonemapping data to the source file before converting from HDR10+ 12 bit to HDR10
This does not require tone mapping. HDR10+ files are always backward compatible with HDR10. When you rescale or reduce bitness from 12 to 10, there's no need to perform tone mapping. Both, the HDR10 (static) as well as the HDR10+ (dynamic, per frame or frames group) will be retained.
Otherwise I can also see a use case for cointaner swapping, something like AV1_12 to HEVC_10?
This would be codec swapping, not container swapping.
I don't think that ffmpeg supports HDR metadata yet with AV1. The spec is pretty new (https://aomediacodec.github.io/av1-hdr10plus/).
Fair enough- in that case I'm not certain.
And yes, HDR meta data in AV1 is not super well supported. per https://www.reddit.com/r/AV1/comments/t5tf2n/script_to_extract_hdr_data_for_svtav1params/
we can use
ffmpeg -i "source.mkv" \
-map 0:V:0 \
-vf crop=3840:1600:0:280,scale=1920:-1:flags=spline \
-c:v libsvtav1 -svtav1-params "preset=6:crf=13:tune=0:enable-hdr=1:color-primaries=9:transfer-characteristics=16:matrix-coefficients=9:mastering-display=G(0.265,0.69)B(0.15,0.06)R(0.68,0.32)WP(0.3127,0.329)L(4000.0,0.005):content-light=10000,424" -pix_fmt yuv420p10le \
-map 0:a:0 -c:a:0 libopus -b:a:0 640k -mapping_family:a:0 1 -filter:a:0 aformat=channel_layouts="7.1|6.1|5.1|stereo|mono" \
-map 0:s:0? -c:s copy \
"encode.mkv"
Problem is here [..]
Please use the command lines I posted. As @FurongZhang had already mentioned: only very specific combinations are working, you are not free to choose whatever you like.
And yes, HDR meta data in AV1 is not super well supported. per reddit.com/r/AV1/comments/t5tf2n/script_to_extract_hdr_data_for_svtav1params
we can use [..]
This is getting a bit off-topic, I think...
I think it's years away before this will become relevant. First there must be a spec, then it takes clients that implement it, then it requires content providers to provide such content, but then - all HDR capable devices will still support HEVC and all content will still be available in HEVC. So why would somebody want to re-encode HEVC-HDR to AV1-HDR or vice-versa? Only when (even later), there might be players that don't support HEVC (maybe due to license cost) or content providers that don't deliver video in HEVC. Re-encoding always involves a quality loss, so you wouldn't want to do such re-encoding "just-for-fun" 😉
Problem is here [..]
Please use the command lines I posted. As @FurongZhang had already mentioned: only very specific combinations are working, you are not free to choose whatever you like.
I'm aware. The problem is getting tonemap_vaapi=format=p010:t=bt2020-10
to h264.
The usecase of tonemap_vaapi is partially to convert a high quality input file to a lower quality, high compatibility file, something like 4k HEVC_10bit HDR @ ~ 50mbps to 2k h264 SDR 5mbps
Per my post above:
When we then pass this to a 10bit capable encoder in a 10 bit capable format (say we use
p010,bt2020,bt2020,bt2020nc
and pass that to hevc_vaapi) we still have the same issue as above, the resulting command and output being:
ffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'tonemap_vaapi=format=p010:primaries=bt2020:transfer=bt2020-10:matrix=bt2020nc' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
The above, with the singular change of primaries to
bt709
with commandffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'tonemap_vaapi=format=p010:primaries=bt709:transfer=bt2020-10:matrix=bt2020nc' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
Which then results in:
tonemap_vaapi=format=p010:primaries=bt2020:transfer=bt2020-10:matrix=bt2020nc
is equivalent to tonemap_vaapi=format=p010:t=bt2020-10
when the primaries are bt-2020 and the matrix are bt2020nc
in this case, perhaps specifying to hevc_vaapi to use 10 bit (instead of relying on autodetect) could fix the issue?
I think it's years away before this will become relevant. First there must be a spec, then it takes clients that implement it, then it requires content providers to provide such content, but then - all HDR capable devices will still support HEVC and all content will still be available in HEVC. So why would somebody want to re-encode HEVC-HDR to AV1-HDR or vice-versa? Only when (even later), there might be players that don't support HEVC (maybe due to license cost) or content providers that don't deliver video in HEVC. Re-encoding always involves a quality loss, so you wouldn't want to do such re-encoding "just-for-fun" wink
Agreed. Supporting it preemptively is a good thing, but that's very much OT
Once I'm setup with things working again, ill try force setting main10 in hevc_vaapi in the command that you commented on:
ffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'tonemap_vaapi=format=p010:primaries=bt2020,scale_vaapi=format=p010' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
would be come something like
ffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'tonemap_vaapi=format=p010:primaries=bt2020,scale_vaapi=format=p010' -c:a copy -c:v hevc_vaapi -profile main10 -b:v 10M -map 0 output/output/sdr_output.mp4 -y
Because of the dead chip I'm having to recompile everything to the limited avx2 ISA, which has been a bit of a nightmare...
None of the above was my command line!
None of the above was my command line!
I miss understood this post then https://github.com/intel/media-driver/issues/1386#issuecomment-1109215274
Here it is again:
ffmpeg -loglevel verbose -y -c:v:0 hevc -hwaccel:v:0 vaapi -hwaccel_device:v:0 /dev/dri/renderD128 -hwaccel_output_format:v:0 vaapi -i hdr_source.mp4 -filter_complex "[v:0]scale_vaapi@f1=w=640:h=360,tonemap_vaapi@f2=format=nv12:matrix=bt709:primaries=bt709:transfer=bt709[f2_out0]" -map [f2_out0] -an -sn -c:v:0 h264_vaapi out.mkv
For hevc output just change h264_vaapi
to hevc_vaapi
.
Given a user with these goals:
use intel i/d GPU with ffmpeg to convert and tonemap an HEVC 10bit HDR DCI 4k file to h264 8bit SDR at DCI 2k, using hardware acceleration for on the fly trans coding, what should that command look like?
using a simple filter, I believe
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'scale_vaapi=w=1920:h=-1,tonemap_vaapi=format=nv12:primaries=bt709:t=bt709' -c:a copy -c:v h264_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
is correct
None of the above was my command line!
I miss understood this post then #1386 (comment)
That was about the HDR-to-HDR tone mapping. I think we got past this already as you don't have a use case for it.
Reposing the question:
Given a user with these goals:
use intel i/d GPU with ffmpeg to convert and tonemap an HEVC 10bit HDR DCI 4k file to h264 8bit SDR at DCI 2k, using hardware acceleration for on the fly trans coding, what should that command look like?
using a simple filter, I believe
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'scale_vaapi=w=1920:h=-1,tonemap_vaapi=format=nv12:primaries=bt709:t=bt709' -c:a copy -c:v h264_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
is correct
This is how it's working (again):
ffmpeg -loglevel verbose -y -c:v:0 hevc -hwaccel:v:0 vaapi -hwaccel_device:v:0 /dev/dri/renderD128 -hwaccel_output_format:v:0 vaapi -i hdr_source.mp4 -filter_complex "[v:0]scale_vaapi@f1=w=640:h=360,tonemap_vaapi@f2=format=nv12:matrix=bt709:primaries=bt709:transfer=bt709[f2_out0]" -map [f2_out0] -an -sn -c:v:0 h264_vaapi out.mkv
None of the above was my command line!
I miss understood this post then #1386 (comment) That was about the HDR-to-HDR tone mapping. I think we got past this already as you don't have a use case for it.
That's what I want to get to next. I can imagine a case where a user is in a limited bandwidth scenario (perhaps wifi on a local nextwork?)
and wants to limit an existing file to something lower such as 10mbps, but the TV/receiver/device has support for the full set of features within the content.
We then need to extract the existing metadata and apply it to a newly computed file.
How would we do this
That's what I want to get to next. I can imagine a case where a user is in a limited bandwidth scenario (perhaps wifi on a local nextwork?)
and wants to limit an existing file to something lower such as 10mbps, but the TV/receiver/device has support for the full set of features within the content.
We then need to extract the existing metadata and apply it to a newly computed file.
How would we do this
Just use a scale_vaapi
filter and nothing else (no tone mapping).
I can't promise that it's working, but that's how it's supposed to work (when it's all fully implemented in ffmpeg).
ffmpeg attaches the dynamic HDR data (and many other SEI data bits) to each frame. This is preserved, even when frames are processed by hardware. Finally, the encoder needs to insert the data into the bitstream.
I suppose we have addressed this issue. Hence, close it. Please feel free to reopen it if any.
System information
CPU information model name : 12th Gen Intel(R) Core(TM) i7-12700K
GPU information:
00:02.0 Display controller [0380]: Intel Corporation AlderLake-S GT1 [8086:4680] (rev 0c)
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] [1002:731f] (rev ca)
Display server if rendering to display(X or wayland): Wayland
Issue behavior
This is a branch of another problem discovered while working on #1342 regarding vaapi hardware accelerated ffmpeg filters on ADL-S.
When using HEVC encoding on HDR10 in the pipeline
ffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'scale_vaapi=w=1920:h=-1,tonemap_vaapi=format=nv12:primaries=bt2020:t=bt709' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
we receive:![image](https://user-images.githubusercontent.com/20584423/163469875-da3771b2-4ab4-447d-8426-df5b290ecaf0.png)
where as using
ffmpeg -loglevel verbose -hide_banner -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i hdr_source.mp4 -vf 'scale_vaapi=w=1920:h=-1,tonemap_vaapi=format=nv12:primaries=bt709:t=bt709' -c:a copy -c:v hevc_vaapi -b:v 10M -map 0 output/output/sdr_output.mp4 -y
creates the correct:
However the filter tonemap_vaapi propagates the input colour primaries (in this case bt 2020, but we must use bt709 to get a usable image out of the encoder)
Describe the current behavior
encoder does not correctly use BT2020 colour primaries
Describe the expected behavior
can correctly use BT2020 colour primaries in tonempaing in combination with HEVC encoder.
issue is not present in vaapi_vp9, vaapi_h264 or any quicksync encoders (including HEVC).
Debug information