YouTube export preset exceeds recommended bitrate by several 100%

[Gizemquea]

Shotcut 19.07.15 Windows 10 (broken on older versions too)

As stated on https://support.google.com/youtube/answer/1722171?hl=en the recommended bitrate for 1080p video @30FPS SDR is 8 Mbit/s (HDR: 10Mbit/s).

When exporting a 1080p video with the YouTube preset, the resulting files are ridiculously big - and the video bitrate exceeds recommendatons by far (up to a factor 10) - regardless of the input material (tried in at least 5 different projects). I am not talking about "spikes" in the VBR stream here, but on the average bitrate of the whole file.

Other settings of the YouTube preset seem to match YouTube's recommendations exactly.

Is this a bug in the codec confugration or simply a wrong quality setting?

After examining the logs of several exports, I think this may be codec dependent e.g. if the global_quality preset 59% is passed to h264_qsv, this results in close to 10 times the desired bitrate, also exceeding the bitrate of the original material by far. In order to obtain a resonabl bitrate of ~10Mbit/s for full HD video with h264_qsv, I have to set Quality to about 10% (and in my case, this actually looks close to what YouTube provides).

None of the few other codecs I briefly tried came even close to YouTube bitrate recommendations.

[ddennedy]

I intentionally ignored their bitrate recommendation. Otherwise I expect people to complain about quality.

[Gizemquea]

I understand your intention, but I would kindly ask you to reconsider this decision for the following two reasons:

1.) I actually ran into serious problems with invalid video being exported using the default YouTube settings i.e. video players would only play the first few seconds of a gigantic 70GB+ export. VLC complained about various issues in the video data stream. The problem only went away once I reduced quality to get about half the original bitrate. I can try to reproduce this and put up an example+project+VLC messages if you're willing to look into this. (I didn't report this as a bug since the YouTube preset video size seemed impractical anyway.)

2.) The YouTube preset takes up even MORE space than H.264 Main Profile, since YouTube recommends GOP 15.

Is it really wise to do that for an export format that is primarily intended for uploading stuff to a video platform? I mean, YouTube will process the uploaded data so that it comes close to their own recommendations - IIRC only this re-processed video is available for download if the uploader ever chose to do that, they don't keep the original data.

Yes, YouTube Video quality is absolute crap for scenes with complex movement, but what is the point in uploading 10 times the data if it doesn't show on the target platform? At 70+GB, size really starts to matter...

If a user wants to export a H.264 video with decent quality for his own archive, isn't that what H.264 Main or High Profile with a GOP of 149 is for?

[ddennedy]

invalid video being exported

Just because VLC has trouble with throughput does not make it invalid. I have seen VLC struggle with 4K UHD or other high bitrate on some systems perhaps depending on hardware accelleration. If you want to check the integrity of a file, open it in Shotcut and choose Properties > menu button > Start Integrity Check Job. When the job completes, double-click the job to view its log. This jobs runs it through ffmpeg to decode it and report errors, for example: ffmpeg -xerror -err_detect +explode -v info -i I:/testing/test.mp4 -f null pipe:

I just made a test from 1080p60 footage using libx264 and it played fine for me in VLC. MediaInfo reports the average bitrate of this as 29 Mb/s, which is not that big IMO. (h264_nvenc gave me 60 Mb/s.) Some organizations use 100 Mb/s for an intermediate or mezzanine, which is essentially what this is. It is fairly well known that YouTube output quality gets better with the most quality you can tolerate to upload.

they don't keep the original data

That is not true; they just do not make it available for download. They have been known to retranscode portions of their library when new codecs such as vp9 or features such as 5.1 surrounds have been added.

what is the point in uploading 10 times the data if it doesn't show on the target platform?

Some people prefer and recommend to upload lossless or visually lossless to YouTube, and you never know what higher bitrates YouTube may add in the future for adapative bitrate streaming. YouTube takes a large variety of inputs, and honestly, I think they created the guidelines just to give users some kind of answer and probably more like a minimum ala system requirements.

Perhaps there can be some compromise, but I cannot switch to ABR because our presets are not smart (if resolution X then bitrate Y etc.). How about 55% = x264 crf 23 per the default? For my test MediaInfo reports 26.5 Mb/s. Somehow, I suspect that is not low enough for you either. 50% gives crf 26 and 17.9 Mb/s. 50% is a nice number.

[Gizemquea]

First of all I'd like to thank you for discussing this in detail. I really appreciate that.

I mentioned VLC because this was the application that I first thought of once Windows Media player and the Windows Movies & TV App were all unable to play the exported file. All three applications stopped at the same time a couple of seconds from the start and were unable to jump to any later point in the video. With VLC I had easy access to at least some kind of diagnostic info.

This happened to me with two independent exports, so I'm fairly sure that I will be able to reproduce the problem and provide output of the integrity check as you suggested. Most probably this is a h246_qsv-specific codec issue that you cannot address directly anyway.

Just give me a few hours to try this out.

Regarding bitrate of the YouTube preset:

As I suspected in my original report, the resulting data rate will depend on the actual codec and h264_qsv seems to generate a very high bitrate. libx264 with 29Mbit/s sounds indeed quite reasonable. This is at least close to the bitrate provided by popular consumer cameras for full HD at 30-60FPS.

Ultimately it comes down to this: unless you apply special filters, overlays or you usually cut fewer frames than the GOP of the original material, it doesn't make any sense if the output bitrate is significantly higher than the bitrate of the input material. Some filters like stabilize will actually reduce the required bitrate if working correctly. In my case, h264_qsv actually triples the bitrate of the input material and I mostly use very long scenes with now special effects whatsoever.

Since the existing YouTube preset gives fairly reasonable bitrates for other codecs, there is not much point in reducing global_quality just because the one codec that I use behaves differently.

There must be a better (codec-specific?) solution. Is there a feasible way to achieve comparable (let's say +/-30%) bitrates for all the codecs?

Regarding quality vs. upload time:

How about adding a separate "YouTube (upload only)" preset which results in bitrates closer to YouTube's recommendations?

[ddennedy]

it doesn't make any sense if the output bitrate is significantly higher than the bitrate of the input material.

I disagree because this is lossy compression, and you are losing a generation when re-encoding.

How about adding a separate "YouTube (upload only)" preset

No, that is confusing. YouTube output is always intended for upload to YouTube. People can reduce quality % or switch to ABR if they want.

Is there a feasible way to achieve comparable (let's say +/-30%) bitrates for all the codecs?

Quality % is simply a mapping to the range for each codec's parameter for quality or quantization. I do not want to try to make them all behave similarly along the scale with respect to bitrate or even quality, which is arguably more logical behavior. Maybe I can extend the preset system to have different parameter values per codec.

[ddennedy]

I just tested h264_qsv on Windows 10 with a 7 minute clip of 1080p30. The average bitrate came out to be 90 Mb/s, which is rather high. Nevertheless, Windows 10 media player and VLC were both able to play it in its entirety. The Shotcut integrity check reports no problem. This is GoPro motorcycle footage; so, it is complex with a lot of motion. Not sure what is going on with yours. Maybe it is an early poor implementation of Quick Sync? I am running a 2018 MacBook Pro with processor Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz, 2601 Mhz, 4 Core(s), 8 Logical Processor(s), Intel(R) Iris(R) Plus Graphics 655.

[Gizemquea]

I just tested h264_qsv on Windows 10 with a 7 minute clip of 1080p30. The average bitrate came out to be 90 Mb/s, which is rather high. Nevertheless, Windows 10 media player and VLC were both able to play it in its entirety. The Shotcut integrity check reports no problem. This is GoPro motorcycle footage; so, it is complex with a lot of motion. Not sure what is going on with yours. Maybe it is an early poor implementation of Quick Sync? I am running a 2018 MacBook Pro with processor Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz, 2601 Mhz, 4 Core(s), 8 Logical Processor(s)

I use a Microsoft Surface Pro 6 (2018) with a Core i5-8250U.

Unfortunately I wasn't able to reproduce the behavior with a trimmed down version of the original project. It played beyond the point where the problem originally occured, right to the end of the ~1 minute test export.

I will have to try another full export tonight (can't do that right now, I'm not at home and I need an external drive for that much data).

There are a few things I can think of: 1.) the problem is somehow related to the file size of the original project export. We'll know as soon as I get to export the whole thing again. 2.) the real cause was data corruption on the external drive (however, simply reading or copying/uploading the file seemed to work fine), If this really turns out to be the case then please accept my apologies for wasting your time.

[Gizemquea]

it doesn't make any sense if the output bitrate is significantly higher than the bitrate of the input material.

I disagree because this is lossy compression, and you are losing a generation when re-encoding.

Yes, you are losing a generation when re-encoding, but does this really justify (in case of h264_qsv) to spend roughly twice the original bitrate on basically trying to replicate the compression artifacts of the previous encoder?

In the special case of h264_qsv we are talking about turning original material of ~30Mbit/s (GoPro footage, either HEVC or H.264) into a 90Mbit/s output (that's what I meant with "significant") and not - let's say - spending an additional 1/3 = 10Mbit/s to compensate for the inevitable quality loss of a re-encoding. Nobody would start a discussion if the export is just 40Mbit/s instead of 30.

The libx264 bitrate of the youtube preset seems just fine in this respect, this really seems to be a codec issue (i.e. how global_quality is translated into codec-specific quality-settings).

[Gizemquea]

I'm afraid I am not able to reproduce the invalid video problem anymore. I exported the whole project again, using the YouTube preset with h264_qsv. The 74GB file seems to play without any issues.

I really tried to get as close to the original conditions as I possibly could, using the same external drive and even the same cable on the same USB hub etc. - but there are some minor changes that I simply cannot undo (in the project itself as well as the exact contents of the external drive).

I'm awfully sorry to have wasted your time on this. I am a software developer myself and I know this can be really annoying, but since this happened twice when I tried to get the original export done, there was really no way I could have known that I would be unable to reproduce this a third time.

One question though: Since I promised to do this ASAP, I didn't let the export run overnight as I usually do. I started the export and just waited until I got a time estimation and came back close to that time, to examine the result. The original estimation of close to 4h was really good, I returned at about 97%.

But then I noticed something strange: the export file seems to grow until the progress reaches 99%, but then progress is stuck at 99% for more than 1h, which means that the real export time is more than 5h.

Once the file doesn't grow anymore, disk activity shows that it is reading AND writing at full speed (old mechanical HDD, so this is supposed to be slow). The original files are on another drive, and it doesn't read anything there. If I calculate the read/write speed and compare that to the file size, it seems that the whole file is read and written "again".

What is going on here? In particular, what is it writing into the growing file until it gets to 99% and why is it necessary to process everything again? I'm not saying that anything is wrong with that, I'm just curious about the reason.

Back to the topic: yes, please, I'd really appreciate if you found a solution to adjust the quality settings of the presets depending on the codec. And even if there is no good way to do this: if I just want to upload stuff to YouTube I can still set the quality myself, that's actually the least of my concerns.

Thanks for your time and patience on this subject. I know that I am using a free product and that the time and effort you spend on developing and maintaining it is certainly not for free. :)

[ddennedy]

Once the file doesn't grow anymore, disk activity shows that it is reading AND writing at full speed

See Export > Advanced > Other> movflags=+faststart

This flags moves the moov atom in the mov or mp4 to the beginning of the file. This atom contains the media metadata used to prepare the decoders. This makes it suitable for progressive download streaming over HTTP. It also makes YouTube able to start processing the file before it is completely uploaded. Unfortunately, FFmpeg only writes the atom at the end and must do this as a post-process. Also, unfortunately, there is not good way to estimate and factor this time into the overall job estimate. The job estimate is not reliable, in general, as soon as your project has complexity that is variable over time.

[Gizemquea]

Thanks for the explanation. The export job estimate actually works surprisingly well for me if I export to a fast drive (+/-10% of the actual time) and without faststart it is as good as it can possibly get. I really was just curious.

[ddennedy]

I decided to change the YouTube preset to x264 default crf 23, which gives 55%. How that 55% translates converts to various hardware encoders will vary, but given their quality is less than x264, I am not comfortable to go less. If a person wants more control on the bitrate level, they should switch to average or constrained rate control. https://github.com/mltframework/mlt/commit/3a2edfa1013d9b8e993fa371afa263d268ff8f7b