Black bar on one side of video (dimensions/crop issue?)

[sethrichards]

I've been updating my transcoding process from using transcode_video to other_transcode, and it's generally giving me better results in smaller files. But I sometimes get a strange issue where the output is strangely cropped with a black bar on the right side, or (in the case of DVD-sourced input) the dimensions of the output file are actually expanded to include a black bar:

If I switch over to the x264 software decoder it doesn't seem to have this problem, only the h265 hardware encoder. This machine is:

• Intel Core i7 3770k CPU
• AMD RX580 GPU
• Ubuntu Mate 22.04LTS

Logs are attached. Any idea what's going on here?

A1_t00-x265-edited.mkv.log

[skj-dev]

@sethrichards Not sure if it's the cause, but you are using a old version of ffmpeg (4.4.2 per the log). It might be worthwhile trying the current release (5.1.2) just to see if it makes a difference.

[sethrichards]

Thanks, I'll have to give it a shot. It looks like my best bet may be to update to Ubuntu 22.10, which looks like it includes ffmpeg 5.1.1. The static binaries here (linked from the ffmpeg download page) don't appear to support hardware encoding, which kind of defeats the purpose. And I'm reluctant to deal with building a custom version from source. I'll report back.

[skj-dev]

If you are of the Docker persuasion, I do make container builds ( https://github.com/ttyS0/docker-other-transcode ). The big caveat there is you seem to be using AMD VAAPI, and that is a platform that I don't target. The QSV image would probably work, since it also includes VAAPI, though I'm not sure if there's anything extra that the AMD hardware encoder might need.

I mention it now, since while the Dockerfiles for the QSV, Nvidia, and software encoder targets might be useful if you ended up wanting to build your own ffmpeg. The advantage being, if you're interested in containers, that it wouldn't matter what packages were installed on your distribution.

Just thought I'd throw that out there as another option. :)

[lisamelton]

@ttyS0 Thanks again! That's a twofer today, sir. 👍 👍

@sethrichards Let us know how it goes because I don't see anything else wrong in that .log file that @ttyS0 didn't already notice.

[sethrichards]

I upgraded my Ubuntu install to 22.10, and fortunately it both comes with ffmpeg 5.1.1 and has a new enough set of libraries that I was able to cleanly build 5.1.2 from source. Unfortunately both of those exhibit the same problem with these videos when using hardware h.265. So I assume the problem is either somewhere in the vaapi layer or in the RX580 itself. I also tried forcing it to use the QSV encoder on my CPU, but I just get an error message. Presumably it's just too old (10 years) at this point.

I'll keep the Docker approach in mind, but I have zero experience there. So it would be more of a learning curve / long term project. Maybe if I get some time over the holidays. In the meantime, this only seems to affect pillarboxed HD rips and DVDs; letterboxed HD videos crop & encode fine so far. So it's not too bad to fall back to x264 software encodes for those (and DVDs encode pretty quickly in SW anyway). I'll follow up here if I make any progress on the issue. Thanks for the quick feedback and suggestions!

[lisamelton]

@sethrichards Dang. I was hoping that a newer ffmpeg would fix everything. Sorry about that.

I am baffled as to why it's only happening on your pillarboxed rips. I have many of those as well but, then again, I don't use VAAPI. So maybe that's it?

[skj-dev]

@sethrichards Unfortunately the Docker approach won't help if it's an issue with the AMD VAAPI implementation. It's basically an easy way to get a newer ffmpeg without have to install or build ffmpeg.

FWIW, one of the reasons I don't build an image targeting AMD is I've never read good things about its hardware transcoding ability. VAAPI is also a bizarre beast, which doesn't help.

Something I probably should have asked before, what does your other-transcode line look like? You mentioned that you were switching over from video-transcode. Any chance you carried over a cropping option that isn't lining up? The current guidance, as I recall, is to not try to crop transcodes, but I don't think that's always been the case.

[lisamelton]

@sethrichards Mr. @ttyS0 's comment just made me realize that we should have been asking you what happens to those rips if you use a software encoder with other-transcode instead of VAAPI? Seriously, try the --x264-cbr option.

[sethrichards]

Sorry, my time to poke at this is pretty intermittent lately. This is my other-transcode command-line: other-transcode --crop auto --hevc --10-bit --eac3-aac --add-audio eng=surround --add-subtitle eng "/home/foo/Videos/Mickeys_Christmas_Carol_USA/A1_t00.mkv"

I tried a bunch of variations this morning. Switching to using the x264 software encoder in other-transcode instead of transcoode-video does result in a good transcode with good dimensions. But I tried sticking with HEVC and

1. using --crop 0:0:0:0 instead of "auto"
2. using --crop 720:480:0:0
3. removing the crop parameter entirely
4. removing the --10-bit parameter since the docs say the AMD encoder only does 8-bit

Unfortunately all of these resulted in the same 768x420 encode with a black bar on the right.

On the plus side, using other-transcode for both the x264 and HEVC encodes will let me clean up my wrapper script quite a bit, so that's nice.

[sethrichards]

Oh, also, I tried removing the encoder param entirely to get a hardware-accelerated h.264 encode. That just errors out immediately (I assume from ffmpeg):

-> other-transcode --eac3-aac --add-audio eng=surround --add-subtitle eng "/home/foo/Videos/Mickeys_Christmas_Carol_USA/A1_t00.mkv"
Verifying "ffprobe" availability...
Verifying "ffmpeg" availability...
Verifying "mkvpropedit" availability...
Finding encoders...
Trying "h264_nvenc" video encoder...
Trying "h264_qsv" video encoder...
Trying "h264_vaapi" video encoder...
Scanning media...
duration = 00:25:44.709
Stream mapping:
 0 = h264_vaapi / 1500 Kbps
 1 = copy
 4 = dvd_subtitle
 9 = subrip
Command line:
ffmpeg -loglevel error -stats -vaapi_device /dev/dri/renderD128 -i /home/foo/Videos/Mickeys_Christmas_Carol_USA/A1_t00.mkv -vsync cfr -map 0:0 -filter:v format\=nv12,hwupload -c:v h264_vaapi -b:v 1500k -color_primaries:v smpte170m -color_trc:v smpte170m -colorspace:v smpte170m -metadata:s:v title\= -disposition:v default -map 0:1 -c:a:0 copy -metadata:s:a:0 title\= -disposition:a:0 default -map 0:4 -c:s:0 copy -disposition:s:0 0 -map 0:9 -c:s:1 copy -disposition:s:1 0 -metadata:g title\= -default_mode passthrough A1_t00.mkv
Transcoding...
Could not write header for output file #0 (incorrect codec parameters ?): Invalid data found when processing input
Error initializing output stream 0:0 -- 
/usr/local/bin/other-transcode: transcoding failed: A1_t00.mkv

[skj-dev]

Ugh ... this certainly smells like a problem with the AMD VAAPI encoder and/or driver. 😞

If you're willing to take the encoding speed penalty, I think you'll also get better results with the software based x264.

[lisamelton]

@ttyS0 I completely agree with both points. 👍 👍

@sethrichards Yeah, I think you're much better off sticking with software-based x264 encoding.

[sethrichards]

Not the answer I was hoping for but it makes sense. Thanks for helping me work through it! Incidentally, the entire reason I was trying this in Linux was that hardware encoding in Windows seemed to work fine except I got occasional garbage frames in the output. So I can't recommend the RX580 for use with other-transcode.