Closed morpheus2n2 closed 3 years ago
Please provide examples and logs and your current settings
And please provide one full conversion with DEBUG logging enabled
https://github.com/mdhiggins/sickbeard_mp4_automator/wiki/Debug-Level-Logging
Heres a log from a file that just finished, cos I am using software encoding a debug log will be about 2 days https://pastebin.com/fkiF42W5
You can try and make a test video using -t, 10
in post opts to make a sample clip
Looks like FFProbe isn't returning the framedata and so its not getting added
Try running the following command on the file and posting the output here
ffprobe -hide_banner -loglevel warning -select_streams v -print_format json -show_frames -read_intervals %+#1 -show_entries frame=color_space,color_primaries,color_transfer,side_data_list,pix_fmt -i filename.mkv
You may also want to try running an updated build of ffprobe to see if that fixes it
D:\Downloads\To Convert>ffprobe -hide_banner -loglevel warning -select_streams v -print_format json -show_frames -read_intervals %+#1 -show_entries frame=color_space,color_primaries,color_transfer,side_data_list,pix_fmt -i Chernobyl.S01E01.1.23.45.2160p.UHD.BluRay.x265.10bit.HDR.DTS-HD.MA.5.1-SWTYBLZ.mkv { [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 4 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 5 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 6 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 7 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 8 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 9 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 10 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 11 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 12 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 13 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 14 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 15 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 16 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 17 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 18 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 19 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 0000015b39d74b00] Could not find codec parameters for stream 20 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options "frames": [ { "pix_fmt": "yuvj444p", "color_space": "bt470bg", "side_data_list": [ { "side_data_type": "QP table data" }, { "side_data_type": "QP table properties" } ] } ] }
https://drive.google.com/file/d/1aoyX1i2Lr6_3Rm3O63DgE0mh5ZDXb4kc/view?usp=sharing Here's the output file from the 10 second test
Does adding -probesize 50M -analyzeduration 100M
to ffprobe change the output?
ffprobe -hide_banner -loglevel warning -select_streams v -print_format json -show_frames -read_intervals %+#1 -show_entries frame=color_space,color_primaries,color_transfer,side_data_list,pix_fmt -probesize 50M -analyzeduration 100M -i filename.mkv
Nope exact same output
D:\Downloads\To Convert>ffprobe -hide_banner -loglevel warning -select_streams v -print_format json -show_frames -read_intervals %+#1 -show_entries frame=color_space,color_primaries,color_transfer,side_data_list,pix_fmt -probesize 50M -analyzeduration 100M -i Chernobyl.S01E01.1.23.45.2160p.UHD.BluRay.x265.10bit.HDR.DTS-HD.MA.5.1-SWTYBLZ.mkv
{
"frames": [
{
"pix_fmt": "yuvj444p",
"color_space": "bt470bg",
"side_data_list": [
{
"side_data_type": "QP table data"
},
{
"side_data_type": "QP table properties"
}
]
}
]
}
D:\Downloads\To Convert>
also if its a help heres the media info os the source file https://pastebin.com/u1xKxNBt and of the output file https://pastebin.com/Vfdj1Jrx
Sorry forgot debug log https://pastebin.com/2b7T64MD
Yeah the ffprobe output is messed up for some reason. Try a different build
{
"frames": [
{
"pix_fmt": "yuv420p10le",
"color_space": "bt2020nc",
"color_primaries": "bt2020",
"color_transfer": "smpte2084",
"side_data_list": [
{
"side_data_type": "Mastering display metadata",
"red_x": "34000/50000",
"red_y": "16000/50000",
"green_x": "13250/50000",
"green_y": "34500/50000",
"blue_x": "7500/50000",
"blue_y": "3000/50000",
"white_point_x": "15635/50000",
"white_point_y": "16450/50000",
"min_luminance": "10/10000",
"max_luminance": "10000000/10000"
},
{
"side_data_type": "Content light level metadata",
"max_content": 0,
"max_average": 0
}
]
}
]
}
It should look like this
Im using a build that was complied yesterday
Ok out of curiosity I ran the same command on a 8-bit file and got
D:\Downloads\To Convert>ffprobe -hide_banner -loglevel warning -select_streams v -print_format json -show_frames -read_intervals %+#1 -show_entries frame=color_space,color_primaries,color_transfer,side_data_list,pix_fmt -probesize 50M -analyzeduration 100M -i Star.Wars.Episode.VII.The.Force.Awakens.2015.2160p.UHD.BluRay.x265-DEPTH.mkv
{
"frames": [
{
"pix_fmt": "yuv420p10le",
"color_space": "bt2020nc",
"color_primaries": "bt2020",
"color_transfer": "smpte2084",
"side_data_list": [
{
"side_data_type": "Mastering display metadata",
"red_x": "34000/50000",
"red_y": "16000/50000",
"green_x": "13250/50000",
"green_y": "34500/50000",
"blue_x": "7500/50000",
"blue_y": "3000/50000",
"white_point_x": "15635/50000",
"white_point_y": "16450/50000",
"min_luminance": "1/10000",
"max_luminance": "10000000/10000"
},
{
"side_data_type": "Content light level metadata",
"max_content": 0,
"max_average": 0
},
{
"side_data_type": "H.26[45] User Data Unregistered SEI message"
},
{
"side_data_type": "H.26[45] User Data Unregistered SEI message"
}
]
}
]
}
D:\Downloads\To Convert>
How ever ALL 10-bit files no matter what it is give me
{
"frames": [
{
"pix_fmt": "yuvj444p",
"color_space": "bt470bg",
"side_data_list": [
{
"side_data_type": "QP table data"
},
{
"side_data_type": "QP table properties"
}
]
}
]
}
So I think this is a ffmpeg problem with 10-bit files
Yeah I'm seeing the same thing on my end, not sure how to get that data without FFProbe returning it
It might just be that the source files you're using are lacking this metadata
Here is a sample file of a 10 bit HDR video that reports that information fine
That's a lot of source files then lol
If they're all coming from the same release groups it may be a problem with how they are creating their releases though, the sample file above shows its not a 10 bit issue
good shout lol
Did you find any evidence that MediaInfo is reading those parameters? I didn't see anything on what you posted or on my end
Which Parameters?
All of the color data that's missing (R, G, B, WP, luminance)
Oh yeah I have never seen that in mediainfo not sure if it actually displays it
Here was a response from Chris Griffith who wrote the article that I used to include all this metadata info regarding your issue
Glad it has helped!
Dolby Vision profile levels are confusing as they come with different information. Profile 5 only comes with Dolby Vision, whereas 8.1 comes with DV + HDR10 information. There is also 8.2 and 8.4 that do not,
https://dolby.force.com/professionalsupport/s/article/What-is-Dolby-Vision-Profile?language=en_US.
The file in that issue is reporting it is a “bt.470bg” color space in uncompressed 4:4:4 chroma (yuvj444p), so it does not have HDR10 information (which is for bt.2020 + yuv420p10le). 4:4:4 chroma is usually reserved for content developers, so odd to see it outside that realm, and I do not personally have experience converting that to it’s HDR10 equivalent.
Its a bit beyond me as well but if the data isn't there then I certainly can't extract it
Also I just noticed a small bug in the code which I've fixed when going over this that the R and G color parameters had been switched so make sure you update
Ok cool, thank you sounds like its just a dodgy release then will find new source and try again 👍
https://codecalamity.com/encoding-uhd-4k-hdr10-videos-with-ffmpeg/
Here's the article for reference
Awesome thank you :D
Hi, as a reminder, I'm using your alternate codecs to encode via GPU for non HDR files and using CPU for HDR files.
Detection of HDR seems to be working fine but for some unknown reasons, some resulting files are getting wrong HDR metadata.
Original file mediainfo: Vidéo ID : 1 OriginalSourceMedium_ID/String : 4113 (0x1011) Format : HEVC Format/Info : High Efficiency Video Coding Nom commercial : HDR10 Profil du format : Main 10@L5.1@High Identifiant du codec : V_MPEGH/ISO/HEVC Durée : 2 h 30 min Débit : 57,5 Mb/s Largeur : 3 840 pixels Hauteur : 2 160 pixels Format à l'écran : 16/9 Type d'images/s : Constant Images par seconde : 23,976 (24000/1001) Im/s Espace de couleurs : YUV Sous-échantillonnage de la chrominance : 4:2:0 (Type 2) Profondeur des couleurs : 10 bits Bits/(Pixel*Image) : 0.289 Taille du flux : 60,3 Gio (79%) Titre : Tenet.2020.IMAX.2160p.BluRay.REMUX.HEVC.DTS-HD.MA.5.1-FGT Langue : Anglais Default : Non Forced : Non Gamme de couleurs : Limited Coordonnées de chromaticité : BT.2020 Caractéristiques du transfert : PQ Coefficients de la matrice : BT.2020 non-constant Support d'origine : Blu-ray MasteringDisplay_ColorPrimaries : Display P3 MasteringDisplay_Luminance : min: 0.0050 cd/m2, max: 4000 cd/m2 MaxCLL : 349 cd/m2 MaxFALL : 86 cd/m2
Here is the resulting mp4 mediainfo: Vidéo ID : 1 Format : HEVC Format/Info : High Efficiency Video Coding Profil du format : Main 10@L5@High Identifiant du codec : hvc1 Identifiant du codec/Info : High Efficiency Video Coding Durée : 1 min 0s Débit : 31,5 Mb/s Largeur : 3 840 pixels Hauteur : 2 160 pixels Format à l'écran : 16/9 Type d'images/s : Constant Images par seconde : 23,976 (24000/1001) Im/s Espace de couleurs : YUV Sous-échantillonnage de la chrominance : 4:2:0 Profondeur des couleurs : 10 bits Type de balayage : Progressif Bits/(Pixel*Image) : 0.158 Taille du flux : 226 Mio (76%) Titre : 4K HDR / 4K HDR Bibliothèque utilisée : x265 3.2+35-acc5d16999e2:[Windows][GCC 9.2.1][64 bit] 10bit Paramètres d'encodage : cpuid=1111039 / frame-threads=3 / numa-pools=8 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=3840x2160 / interlace=0 / total-frames=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=1 / no-allow-non-conformance / repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / keyint=250 / gop-lookahead=0 / bframes=3 / b-adapt=0 / b-pyramid / bframe-bias=0 / rc-lookahead=5 / lookahead-slices=8 / scenecut=0 / hist-scenecut=0 / radl=0 / no-splice / no-intra-refresh / ctu=32 / min-cu-size=16 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / no-signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=2 / limit-refs=0 / no-limit-modes / me=0 / subme=0 / merange=57 / temporal-mvp / no-frame-dup / no-hme / no-weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / no-sao / no-sao-non-deblock / rd=2 / selective-sao=0 / early-skip / rskip / fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=abr / bitrate=30000 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 / pbratio=1.30 / aq-mode=1 / aq-strength=0.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=5 / chromaloc=0 / display-window=0 / cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / hist-threshold=0.01 / no-opt-cu-delta-qp / no-aq-motion / no-hdr10 / no-hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 / analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00 / no-scenecut-aware-qpconformance-window-offsets / right=0 / bottom=0 Langue : Anglais Gamme de couleurs : Limited Coefficients de la matrice : BT.470 System B/G Menus : 12
This test was done with the 9/12/2020 FFMPEG release and with the latests version of your script (using dynamic parameters). I was wondering if this is related to the above discussion ? Is this also an issue with the source file not being properly HDR tagged ? I'm having this issue with multiple files from different releases.
Sometimes HDR metadata are properly transferred. If I compare Mediainfo for files that did work with others that don't, I see no difference, so not sure if I can troubleshoot this or if I'm doing something wrong.
For reference, here is my autoprocess.ini file: [Converter] ffmpeg = C:\Script\ffmpeg\ffmpegPR.exe ffprobe = C:\Script\ffmpeg\ffprobePR.exe threads = 0 hwaccels = dxva2, cuvid, qsv, d3d11va, vaapi hwaccel-decoders = h264_cuvid, mjpeg_cuvid, mpeg1_cuvid, mpeg2_cuvid, mpeg4_cuvid, vc1_cuvid, hevc_qsv, h264_qsv, hevc_vaapi, h264_vaapi hwdevices = vaapi:/dev/dri/renderD128 hwaccel-output-format = vaapi:vaapi output-directory = output-format = mp4 output-extension = mp4 temp-extension = minimum-size = 0 ignored-extensions = nfo, ds_store copy-to = move-to = delete-original = False sort-streams = True process-same-extensions = False force-convert = False post-process = False wait-post-process = False detailed-progress = True preopts = -ss,00:10:00,-t,00:01:00 postopts = -max_muxing_queue_size, 9999 bypass-if-copying-all = False
[Permissions] chmod = 0644 uid = -1 gid = -1
[Metadata] relocate-moov = True full-path-guess = True tag = True tag-language = fr download-artwork = poster sanitize-disposition = True
[Video] codec = hevcqsv max-bitrate = 30000 crf = -1 crf-profiles = preset = codec-parameters = dynamic-parameters = True max-width = 0 profile = main10 max-level = 0.0 pix-fmt = yuv420p10le filter = force-filter = False
[HDR] pix-fmt = yuv420p10le space = bt2020nc transfer = smpte2084 primaries = bt2020 preset = ultrafast codec-parameters = filter = force-filter = False codec = h265
[Audio] codec = ac3 languages = eng,fra default-language = first-stream-of-language = False allow-language-relax = True channel-bitrate = 256 max-bitrate = 0 max-channels = 0 prefer-more-channels = True default-more-channels = True filter = force-filter = False sample-rates = copy-original = True aac-adtstoasc = False ignore-truehd = mp4, m4v ignored-dispositions = unique-dispositions = False
[Universal Audio] codec = aac channel-bitrate = 256 first-stream-only = False move-after = False filter = force-filter = False
[Subtitle] codec = srt codec-image-based = languages = eng,fra default-language = first-stream-of-language = False encoding = burn-subtitles = False burn-dispositions = download-subs = False download-hearing-impaired-subs = False download-providers = embed-subs = False embed-image-subs = False embed-only-internal-subs = False filename-dispositions = ignore-embedded-subs = False ignored-dispositions = unique-dispositions = False attachment-codec =
MediaInfo isn't helpful because it doesn't display the frame data. Can you post the logs of the conversion and the ffprobe frame data from the command provided above?
Reference
ffprobe -hide_banner -loglevel warning -select_streams v -print_format json -show_frames -read_intervals %+#1 -show_entries frame=color_space,color_primaries,color_transfer,side_data_list,pix_fmt -i filename.mkv
Hi,
Here is the result of the FFPROBE command on one of the file:
{
"frames": [
{
"pix_fmt": "yuvj444p",
"color_space": "bt470bg",
"side_data_list": [
{
"side_data_type": "QP table data"
},
{
"side_data_type": "QP table properties"
}
]
}
]
}
Converting will require a long time. Do you still need the logs of a conversion ? shall I specify "debug mode" for the logs ?
Nah that's enough info. That ffprobe output is missing the color data so that's why you're not getting it, your source doesn't have the metadata
Ok thanks ! I got it.
I'm manually testing every file before import with the ffprobe command line, but this is manual......
Could you implement, something like maybe a test of the result of the above command line and, if not as expected, you just skip this file and move forward to next item in queue. You could also create an empty text file with the movie name and adding missing.HDR.metadata to it so we know why the file was skipped....digging in the logs would be a pain......
Going back to standard codec (not the HDR one) and encode the file anyway would end up with a washed-out colors movie....so that's not a good option.
It's such a waste of time to encode a file via cpu for nothing.......or even via gpu......if you end up having bad colors......
Thanks anyway
Yeah I'll see if I can come up with something. Wish there was a better solution and I'm open to suggestions
Challenging to solve this one
I have just resorted to skipping all 10bit media as this is the only play I have a problem, if its 8bit it seems to work perfectly fine and keep HDR stuff
I would say that recommendation isn't great since the whole conclusion of this post is that its not a 10-bit problem but a specific format and metadata problem
I'm gonna add a method that returns a boolean value for if metadata is present or not and then people can choose to use that to skip files I think
👍 Cool
I can be your guinea pig !
09e9b320eaa0ebcfbe237eddf1e2f1cd886aa8be
https://github.com/mdhiggins/sickbeard_mp4_automator/wiki/Custom-Source-Validation
Alright here's an update that should help
Big change with this update is that the isValidSource method of MediaProcessor now has a custom code check. If you add a custom.py file to your config folder with a method validation
that returns True or False that method will be included to allow users to run custom code to determine if a file should be considered valid. That wiki page shows the parameters
I also added a sample to the setup folder called config.py.sample
which should allow you to exclude HDR files if they're missing valid color info. You can modify this however you like but that should get you started
Just tested it on a manual conversion and it worked as expected ! File without proper HDR metadata was skipped: C:\Git>python C:\Script\PrePROD\manual.py -a -i c:\test.mkv Manual processor started. Python 64-bit 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)]. Guessit version: 3.1.0. C:\Program Files\Python38\python.exe Loading config file C:\Script\PrePROD\config\autoProcess.ini. Initiating custom validation method. HDR video stream detected for 0. ValidFrameData is False File c:\test.mkv is not in a valid format
I still need to test this on an automatic batch but sounds promising !!!!!! Will let you know if I find out anything.
Thanks !!!!!!!
After all that work we did a few months back, I have found that 10-bit media don't seem to copy over the HDR tagging, I am not sure if its a setting in the autoprocess.ini i need to set or what, any media in HDR that is NOT 10-bit worked perfect every time