Help Requested - 4k UHD video transcoding h265.videotoolbox on macOS

[codethought]

Greetings.. I'm having a hard time figuring out the right set of parameters to get other-transcode and ffmpeg to play nice on transcoding a video. Here's the stats for the video via MakeMKV rip and MediaInfo.

Video
ID                                       : 1
ID in the original source medium         : 4113 (0x1011)
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@High
HDR format                               : SMPTE ST 2086, HDR10 compatible
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 1 h 56 min
Bit rate                                 : 50.6 Mb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0 (Type 2)
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.254
Stream size                              : 41.1 GiB (87%)
Language                                 : English
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : Display P3
Mastering display luminance              : min: 0.0050 cd/m2, max: 1000 cd/m2
Maximum Content Light Level              : 1000 cd/m2
Maximum Frame-Average Light Level        : 512 cd/m2
Original source medium                   : Blu-ray

Here's the problem I'm having... running other-transcode with the following commands works.. but the end result is less than satisfactory..

other-transcode --position 00:10:00 --duration 00:10:00 --add-subtitle eng --burn-subtitle auto --add-audio eng --eac3 --no-filters "/Volumes/MultimediaISO/MakeMKV/MyMovie_t00.mkv"

the ffmpeg command created:

ffmpeg -loglevel error -stats -ss 600 -t 600 -i /Volumes/MultimediaISO/MakeMKV/MyMovie_t00.mkv -map 0:0 -filter:v scale\=1920:1080 -c:v h264_videotoolbox -b:v 6000k -color_primaries:v bt2020 -color_trc:v smpte2084 -colorspace:v bt2020nc -metadata:s:v title\= -disposition:v default -map 0:1 -c:a:0 eac3 -metadata:s:a:0 title\= -disposition:a:0 default -map 0:2 -c:a:1 eac3 -ac:a:1 2 -metadata:s:a:1 title\= -disposition:a:1 0 -map 0:5 -c:a:2 eac3 -ac:a:2 2 -metadata:s:a:2 title\= -disposition:a:2 0 -map 0:6 -c:s:0 copy -disposition:s:0 0 -map 0:7 -c:s:1 copy -disposition:s:1 0 -metadata:g title\= -default_mode passthrough MyMovie_t00.mkv

This works.. however the bit-depth gets transcoded from 10-bit to 8-bit and the resolution gets reduced to 1080p from 2160p. Needless to say not the result I was expecting.

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Format settings, GOP                     : M=1, N=12
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 10 min 0 s
Bit rate mode                            : Variable
Bit rate                                 : 6 016 kb/s
Maximum bit rate                         : 768 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.121
Stream size                              : 430 MiB (87%)
Writing library                          : Lavc58.134.100 h264_videotoolbox
Language                                 : English
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant

Trying to use h264 with the VideoToolbox and 10bit (given the source is 10bit - and I've read 10bit to 10bit should work) fails..

other-transcode --position 00:10:00 --duration 00:10:00  --10-bit --add-subtitle eng --burn-subtitle auto --add-audio eng --eac3 "/Volumes/MultimediaISO/MakeMKV/MyMovie_t00.mkv"
Verifying "ffprobe" availability...
Verifying "ffmpeg" availability...
Verifying "mkvpropedit" availability...
Finding encoders...
Trying "h264_videotoolbox" video encoder...
Scanning media...
duration = 00:10:00
Stream mapping:
 0 = h264_videotoolbox / 6000 Kbps
 1 = eac3
 2 = eac3 / stereo
 5 = eac3 / stereo
 6 = hdmv_pgs_subtitle
 7 = hdmv_pgs_subtitle
Command line:
ffmpeg -loglevel error -stats -ss 600 -t 600 -i /Volumes/MultimediaISO/MakeMKV/MyMovie_t00.mkv -map 0:0 -filter:v scale\=1920:1080 -c:v h264_videotoolbox -pix_fmt:v p010le -b:v 6000k -profile:v main10 -color_primaries:v bt2020 -color_trc:v smpte2084 -colorspace:v bt2020nc -metadata:s:v title\= -disposition:v default -map 0:1 -c:a:0 eac3 -metadata:s:a:0 title\= -disposition:a:0 default -map 0:2 -c:a:1 eac3 -ac:a:1 2 -metadata:s:a:1 title\= -disposition:a:1 0 -map 0:5 -c:a:2 eac3 -ac:a:2 2 -metadata:s:a:2 title\= -disposition:a:2 0 -map 0:6 -c:s:0 copy -disposition:s:0 0 -map 0:7 -c:s:1 copy -disposition:s:1 0 -metadata:g title\= -default_mode passthrough MyMovie_t00.mkv
Transcoding...
[hevc @ 0x7f85ae84ce00] First slice in a frame missing.
    Last message repeated 6 times
[hevc @ 0x7f85ae857e00] First slice in a frame missing.
    Last message repeated 6 times
[h264_videotoolbox @ 0x7f85ae48c000] [Eval @ 0x7ffee06b8b00] Undefined constant or missing '(' in 'main10'
[h264_videotoolbox @ 0x7f85ae48c000] Unable to parse option value "main10"
[h264_videotoolbox @ 0x7f85ae48c000] Error setting option profile to value main10.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
/usr/local/bin/other-transcode: transcoding failed: MyMovie_t00.mkv

I've tried to get other-transcode to leverage the h265_videotoolbox encoder.. but nothing I try seems to work.. I can make the 10bit software encoder work.. but it takes an hour to transcode 10 minutes of video...

As an aside.. I did try to use Handbrake and got the H265 VideoToolbox encoder to work.. and it did.. but the end result was 10-bit to 8-bit transcoding..

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main@L5@Main
HDR format                               : SMPTE ST 2086, HDR10 compatible
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 13 min 38 s
Width                                    : 3 838 pixels
Height                                   : 1 606 pixels
Display aspect ratio                     : 2.40:1
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : Display P3
Mastering display luminance              : min: 0.0050 cd/m2, max: 1000 cd/m2
Maximum Content Light Level              : 1000
Maximum Frame-Average Light Level        : 610

So.. what am I misunderstanding in all of this?

My system is a Mac Pro 6,1 - with an eGPU running an AMD RX580.

Thanks in advance.

[skj-dev]

I think the issue is hardware related.

Looking at this:

    --10-bit, --no-10-bit
                    use 10-bit color depth (default: not used for H.264,
                      used for HEVC with Nvidia, Intel and x265 encoders)

10-bit isn't used for H.264, so that explains why the first one produced 8-bit.

It's a Mac, so Nvidia is off the table.

The MacPro Xeon processors do not have the Intel iGPU in them, so there's no QuickSync (QSV) support, which is what the Intel piece in the above snippet is referring to.

Lastly, there's the software encoder, which was able to work, though as you noted, it's not exactly fast.

Since video_toolbox is the only way to talk to the GPU layer with macOS, there's no way to specifically target the AMD GPU for transcoding. If you were running Linux, then you might be able to coax something out of the AMD GPU, but even then the VAAPI support with AMD GPUs is not great.

Sadly, 10-bit hardware accelerated encoding really needs either an Nvidia card, or an Intel GPU (i.e. consumer grade Intel CPU with built in iGPU).

(And to add insult to injury, 4K transcoding in general remains a dark art largely thanks to the mess that is called HDR.)

[codethought]

Ok.. so if I understand you correctly the reason that Handbrake does do the h265 Hardware Videotoolbox encoding and converts to 8-bit from 10-bit is because that's the closest I can get to the result I want given my hardware.. (and someone in the Handbrake team knows some kind of dark arts). Fair enough... but that does leave me one question.. why doesn't ffmpeg do that if Handbrake does? I mean.. it's academic at this point.. but still leaves me curious.. Or is it that the transcode script logic won't because ffmpeg can't...?

Anyway.. sorry I don't mean to prattle.. it's just a fascinating space for me.

[skj-dev]

To be fair, the dark arts bit really refers to the application of HDR to a 4K title. As I understand it, the transcoding process doesn't preserve any HDR, and it has to be reapplied as a post-transcoding process. The trouble comes from HDR being something that's not easy to either extract, or reapply. I don't yet have anything in the 4K space, so I'm mostly basing that on remembered bits of conversation.

As to the whys and wherefores of the differences between ffmpeg and Handbrake, I really have no idea.

[lisamelton]

@ttyS0 Thanks so much for stepping up to comment here while I was away! 👍

@codethought The other-transcode tool intentionally limits the output size of H.264 output to fit within a 1920 by 1080 pixel boundary. This is because various hardware encoders don't support larger sizes for H.264 output.

Does adding --hevc to your other-transcode command line allow you to produce HEVC output via hardware on your Mac?

[codethought]

@donmelton This is what happens when I try --hevc

other-transcode --hevc  --add-subtitle eng --burn-subtitle auto --add-audio eng --eac3 "/Volumes/MultimediaISO/MakeMKV/XXX_t00.mkv"
Verifying "ffprobe" availability...
Verifying "ffmpeg" availability...
Verifying "mkvpropedit" availability...
Finding encoders...
Trying "hevc_videotoolbox" video encoder...
Scanning media...
duration = 01:56:22.656
Stream mapping:
 0 = hevc_videotoolbox / 8000 Kbps
 1 = eac3
 2 = eac3 / stereo
 5 = eac3 / stereo
 6 = hdmv_pgs_subtitle
 7 = hdmv_pgs_subtitle
Command line:
ffmpeg -loglevel error -stats -i /Volumes/MultimediaISO/MakeMKV/XXX_t00.mkv -map 0:0 -c:v hevc_videotoolbox -b:v 8000k -color_primaries:v bt2020 -color_trc:v smpte2084 -colorspace:v bt2020nc -metadata:s:v title\= -disposition:v default -map 0:1 -c:a:0 eac3 -metadata:s:a:0 title\= -disposition:a:0 default -map 0:2 -c:a:1 eac3 -ac:a:1 2 -metadata:s:a:1 title\= -disposition:a:1 0 -map 0:5 -c:a:2 eac3 -ac:a:2 2 -metadata:s:a:2 title\= -disposition:a:2 0 -map 0:6 -c:s:0 copy -disposition:s:0 0 -map 0:7 -c:s:1 copy -disposition:s:1 0 -metadata:g title\= -default_mode passthrough XXX_t00.mkv
Transcoding...
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
/usr/local/bin/other-transcode: transcoding failed: XXX_t00.mkv

[jaromirrivera]

This use to work in macOS Big Sur. I was able to create 10-bit h265 encodes with hardware acceleration (via VideoToolBox using Intel QuickSync) using other-transcode.

I have since upgraded to macOS Monterey and cannot get any 10-bit hw accelerated h265 encodes to work...

other-transcode --hevc --vt ../NFL\ Football\ \(2002\)\ -\ 2021-09-09\ 08\ 00\ 00\ -\ Dallas\ Cowboys\ at\ Tampa\ Bay\ Buccaneers.t
Verifying "ffprobe" availability...
Verifying "ffmpeg" availability...
Verifying "mkvpropedit" availability...
Finding encoders...
Scanning media...
duration = 03:54:12.0115
Stream mapping:
 0 = hevc_videotoolbox / 4000 Kbps
 1 = copy
Command line:
ffmpeg -loglevel error -stats -i ../NFL\ Football\ \(2002\)\ -\ 2021-09-09\ 08\ 00\ 00\ -\ Dallas\ Cowboys\ at\ Tampa\ Bay\ Buccaneers.ts -vsync cfr -map 0:0 -filter:v yadif -c:v hevc_videotoolbox -b:v 4000k -color_primaries:v bt709 -color_trc:v bt709 -colorspace:v bt709 -metadata:s:v title\= -disposition:v default -map 0:1 -c:a:0 copy -metadata:s:a:0 title\= -disposition:a:0 default -sn -metadata:g title\= -default_mode passthrough NFL\ Football\ \(2002\)\ -\ 2021-09-09\ 08\ 00\ 00\ -\ Dallas\ Cowboys\ at\ Tampa\ Bay\ Buccaneers.mkv
Transcoding...
[hevc_videotoolbox @ 0x7fd7121efe00] Error encoding frame: -12905
[hevc_videotoolbox @ 0x7fd7121efe00] popping: -542398533
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

[zcutlip]

Just stumbled across this issue, and sharing what I've learned.

On my 2018 intel Mac mini, if I generate an ffmeg command with --x265 and --dry-run, and then substitute -c:v hevc_videotoolbox for -c:v libx265 in the resulting command, ffmpeg runs without complaint (I haven't looked at the results yet, still experimenting)

It seems like other-transcode is omitting some arguments that the hevc_videotoolbox encoder requires. Here's a comparison of the -map 0:0 portion of the two ffmpeg commands for the video stream. I've vertically aligned the arguments to make it more clear which are missing:

-map 0:0 -c:v hevc_videotoolbox                        -b:v 8000k                                     -color_primaries:v bt2020 -color_trc:v smpte2084 -colorspace:v bt2020nc -metadata:s:v title\= -disposition:v default
-map 0:0 -c:v libx265           -pix_fmt:v yuv420p10le -b:v 8000k -maxrate:v 25000k -bufsize:v 25000k -color_primaries:v bt2020 -color_trc:v smpte2084 -colorspace:v bt2020nc -metadata:s:v title\= -disposition:v default

It appears that the following options are being omitted when hevc_vdeotoolbox is in use:

• -pix_fmt:v yuv420p10le
• -maxrate:v 25000k
• -bufsize:v 25000k

I don't know enough about ffmepg or its various encoders to know what arguments are required when, but when I include all those, hevc_videotoolbox seems to run fine.

Hope this helps.

[zcutlip]

One more observation: if I pass --10-bit this causes the -pix_fmt argument to be set which seems to satisfy ffmeg.

So where this doesn't work, resulting in @jaromirrivera's error:

other-transcode ../The\ Wire\ Season\ 4\ Disc\ 2\ title_t00.mkv --hevc --vt

This does seem to work (again, haven't checked output):

other-transcode ../The\ Wire\ Season\ 4\ Disc\ 2\ title_t00.mkv --hevc --vt --10-bit

[lisamelton]

@zcutlip Thanks for the detailed information from your own testing! Yes, it appears the --10-bit option is required on older Macs.