Use cases for video_transcoding vs other_video_transcoding

[crerwin]

Hi there, I'm curious about the intended use cases for the video_transcoding gem and the other_video_transcoding gem. Is other-transcode intended as a replacement for transcode-video, or are they for different purposes?

My understanding is that the original intent of transcode-video was to produce small output files that still retain visual quality (with non-default options to sacrifice file size for additional quality), and the intent of other-transcode is to leverage hardware transcoders while avoiding the normal pitfalls of hardware transcoding, but it's not quite clear if the output produced by other-transcode is expected to have the same quality as transcode-video.

I'm ripping my physical library and I want to save the highest quality files I can. If ripping a blu ray results in an h264 or h265 file, it goes right to Plex. If it's a VC-1 file or mpeg2 or anything else that's not widely supported by Plex clients, I'd like to transcode to h264 or h265 with whatever process would allow me to retain the most quality. I'd rather spend more compute time transcoding and ending up with a larger, higher-quality file in the end.

I have a Kubernetes cluster running on some old hardware and I've been using transcode-video to churn through a queue of ripped files, but I'm wondering if other-transcode might be a better option?

Thanks! And thank your for both projects!

[khaosx]

Howdy @crerwin, and thanks for the question. I'll throw in my opinion and @donmelton can chime in to correct anything I leave out or get wrong.

The overall use case is the same for both projects. The aim is to produce files that are as close to visually indistinguishable from source as possible, while decreasing size and increasing portability. Those are the three objectives of the project: fidelity, size, portability. As you pointed out, the engines behind each are different, with other-transcode using various flavors of hardware encoding to increase the speed of the transcode.

We've been testing, reviewing, and refining the process by which other-transcode operates, and, in my opinion, we've hit some very high benchmarks in terms of the three objectives. Personally, I've found that HEVC --10-bit --nvenc-temporal-aq works superbly with my GPU (nVidia 1660ti). Your particular GPU (external or integrated) will dictate what your own settings would be. I get ~130 fps converting full Blu-Ray rips to HEVC, with quality that is on par with the source, viewed on both a 65' OLED panel and projected to a 106" screen.

I started my use of transcode-video just to reduce the size of my kid's videos, and, like you, I just copied out a lot of my movies to storage with no transcoding at all because I wanted to be as close to source as I could get. I'm now at the point where I trust the project's output so much that I don't allow any media that hasn't been transcoded with other-transcode into my library.

Don't take my word for it though. Rip "Saving Private Ryan (1998)" and "Blade Runner 2049 (2017)" and judge for yourself. They are the gold standards for "how does this look?", especially chapter 9 in "Saving Private Ryan (1998)". When you find a workflow that makes that scene look good, you can be satisfied that the vast majority of your encodes will look great.

At the end of the day, you'll be well-served by either of the tools. other-transcode just makes it significantly faster to rip through your library. If you have the ability to do the setup, I recommend it. We also have users that can speak to the viability of throwing K8s into your workflow, but it's out of my area of expertise.

[skj-dev]

I have a Kubernetes cluster running on some old hardware and I've been using transcode-video to churn through a queue of ripped files, but I'm wondering if other-transcode might be a better option?

It really depends on what "some old hardware" means. 😉

If you have Skylake or newer Intel CPUs in your Kubernetes worker nodes, then you might want to take a look at the Docker container I put together for other-transcode (https://github.com/ttyS0/docker-transcode-vaapi). You'll need to make sure you have the intel-gpu-plugin running as a DaemonSet (this is the config I use: https://github.com/ttyS0/kubernetes/blob/master/kube-system/intel-gpu-plugin.yaml). If you're planning on having multiple encodes running at the same time, you'll want to use Pod AntiAffinity to make sure multiple Pods don't land on the same worker node, as mapping the /dev/dri is a locking event, so it can only work with one Pod per node.

Having said all of that, I don't actually use other-transcode on my K8s cluster just because for me "some old hardware" means i3 Kaby Lake NUCs and I don't really have enough headroom to handle batch encoding. One day I hope to get some better hardware in there and setup some kind of transcoding pipeline system as you allude to. 😄

I do, however, run Plex on my K8s cluster (https://github.com/ttyS0/kubernetes/blob/master/plex/plex-server.yaml), and use the /dev/dri passthrough from the intel-gpu-plugin to provide Intel iGPU acceleration. The relevant config piece in there is:

resources:
  limits:
    gpu.intel.com/i915: 1

Also, one of these days I'm planning on getting a worker node with an NVidia card in it so I can have a corresponding Docker container for HEVC encoding. That's totally dependent on me getting hardware, though, and since I just did a home lab migration project I suspect there won't be any new significant hardware coming in until next year. 😄

[lisamelton]

@crerwin Thank you so much for your excellent question! And thanks to @khaosx and @ttyS0 for your wonderful (and correct) comments. I should sleep in more often. :)

The only thing I would add is to point out the big difference in the underlying tools used by the two projects and the video transcoding ratecontrol systems available to those tools.

My older transcode-video tool is based on HandBrakeCLI. And while hardware video encoders are available within HandBrake, it's not possible to tune the ratecontrol systems in those encoders, specifically those available on the Windows platform, to produce the best quality.

But my other-transcode tool is based on ffmpeg, which has access to not only more hardware, but lots of "knobs" with which to tune that hardware. And because I didn't want to retrofit and/or complicate transcode-video and the other tools within the video_transcoding Gem, I decided to create a whole new project.

It's true, as @khaosx so clearly explained, that the other_video_transcoding Gem has exactly the same goals as the older project. It's just the means to achieve those goals have changed. My apologies for the confusion.

Now, I won't tell you which tool you should be using, but you should know that @khaosx, @ttyS0, myself and the rest of my collaborators have all re-done our video libraries with other-transcode. :)

If you would like to join us in discussing (and collaborating on) the project using Slack, I can send you an invite to the workspace and channels I set up about a year ago. Just send me a request via email (because I don't want you to include your real email address here). You can find my email address here:

https://donmelton.com/contact/

Thanks again!

[crerwin]

Thank you all for the quick and thorough responses! I think another way I could have phrased my question is: "Is there a downside to using other-transcode instead of transcode-video?" and it seems like there is not. If other-transcode is at worst almost as fast as transcode-video and gets faster if hardware transcoding is available, then that gets me what I need as far as one tool for my workflow. I plan to run transcoding jobs on a few sets of hardware and not having to customize the command for the underlying hardware is hugely attractive.

It really depends on what "some old hardware" means. 😉

Good point! Current nodes are Xeon 5660s with no GPUs, so I don't expect any hardware transcoding, but if I can swap transcode-video out for other-transcode with no negative effects on output quality, that lets me take advantage of other hardware in the future without changing my workflow.

I do, however, run Plex on my K8s cluster...

Awesome! I had some issues getting Plex running in Kubernetes previously so I will definitely take a look at your info!

If you would like to join us in discussing (and collaborating on) the project using Slack, I can send you an invite to the workspace and channels I set up about a year ago. Just send me a request via email (because I don't want you to include your real email address here). You can find my email address here:

Will do, thank you!

[buffaloseven]

Related to this, if I am to primarily encode my videos on macOS using a 2018 rMBP w/560X and RX 580 in an eGPU, are there any particular flags I want to set while encoding using other_video_transcoding?

[lisamelton]

@buffaloseven Good question. The good news is that you don't need to do anything special to get good quality video from the other-transcode tool on macOS. You can probably even get decent 8-bit HEVC output with that hardware. But...

The bad news is that I can't do anything to tune the video hardware ratecontrol systems on macOS to further enhance quality. This is because the Video Toolbox API, which ffmpeg has to use on macOS, is a black box. There are no "knobs" to turn other than setting the target bitrate.

This is why the recommended platform for using other-transcode is Windows. You can read more about that rationale here:

https://github.com/donmelton/other_video_transcoding/wiki/Platforms

[buffaloseven]

Thanks for the response! So for optimal encoding I’d want to do it on a Boot Camp partition. If I want to move into HEVC (assuming Windows), will other-transcode pick the best for my setup automatically with the —hevc flag, or should I be manually setting the encoder?

And only other question; I did an 8-bit HEVC encode of Good Will Hunting off an old DVD last night (macOS), and it looks pretty good. The file plays everywhere except in Catalina’s TV app. I’ll chalk it up to a bug on their end? Or is there some secret comparability flag I should be setting?

[lisamelton]

@buffaloseven Sorry I took so long to reply. Lots of distraction going on right now.

I'm not sure you can access hardware encoders in Windows when running Boot Camp. But if you try that and it works, please let us know.

Yes, the other-transcode tool will automatically choose 10-bit encoding when the right hardware is available if you use the --hevc option.

I'm not sure why your 8-bit HEVC transcoding isn't working in Catalina's TV app. That seems odd.

[willie]

To sum up what I've read in this thread: if you are using a recent Windows computer, use other-transcode. Otherwise if you are on Mac, use transcode-video.

Is that right, @donmelton?

[lisamelton]

@willie No, I wouldn't put it that way. On macOS, there's essentially no difference in the output from other-transcode and using, for example, the vt_h264 encoder with transcode-video. It's just that using other-transcode makes it easier because that encoder choice is automatic. Plus, there have been other usage improvements with other-transcode because I've taken the opportunity to both simplify the set of command line options and at the same time make them more powerful.

[willie]

@donmelton Thanks for the clarification. I appreciate it! I like your defaults, so I'll stick with other-transcode for now since it is much faster on my Macs.

[TehMaster]

Hey @donmelton - This thread highlighted another question i have - If i am not doing HVEC encodes, and was already using --avbr with transcode-video, is there a benefit to now using other-transcode? 10Bbit HVEC is much slower on my system (only has a nvidia 640 GPU) then software using cpu (AMD Ryzen 3900X - 12 core). Would i see a huge benefit if i swapped in my 1080ti and really started pumping out HVECs?

Plain and simple: Is 10bit HVEC superior to the x264 produced by transcode-video enough to move around some hardware (which i have) and change processes?

Several of you have mentioned you redid your libraries using other-transcode, and now i am stressing out that i should do the same. Maybe i need to do several different encodes and really study the output, but i trust the opinions held here by much of the community.

[lisamelton]

@TehMaster Yes, 10-bit HEVC output from Nvidia or Intel hardware using other-transcode is superior in quality to 8-bit H.264 output from the x264 software encoder using my AVBR algorithm. This is because 10-bit HEVC significantly reduces, and in some cases eliminates, color banding.

Myself, @khaosx, @ttyS0 and several others who haven't commented on this issue yet have all re-transcoded our entire media libraries with other-transcode and various Nvidia GPUs to get 10-bit HEVC. It's that good.

Also, you should note that although the default target bitrate for 1080p content from other-transcode when output as 10-bit HEVC is 6000 Kbps, you can lower that significantly without loss of quality for many types of input, particularly TV shows. I'll let @khaosx comment, if he so desires, because he's had experience with that.

[TehMaster]

@donmelton As always i sincerely appreciate the detailed response. Up until your post yesterday (before i knew of other-transcode) i was happily going through my life with daily use of transcode-video. Now my world has been rocked, and i feel a large re-transcoding project on the horizon... Luckily we have all this time on our hands huh?

I often do transcode my TV shows down to around 3000, and tested using 'other-transcode --hevc --10-bit --target 3000 --eac3' with good output results, but would highly enjoy hearing from @khaosx and his methods.

[khaosx]

@TehMaster Funny, I was just talking to @donmelton yesterday about this very topic. I noticed that I hadn't re-encoded a bunch of my shows, notably Doctor Who, and all of the CW superhero shows (don't judge). I did some testing and determined that, for my environments, a target of 4000 Kbps is my sweet spot. 3000 Kbps is good enough for most applications, but at 4000 Kbps I cannot tell any difference between the encode and the original on either an OLED display or my projector. Weirdly, every single "hour long" show I do reports back at 1.4GB an episode, which is fine for me. It freed about half a TB, which was welcome indeed.

My command line is --eac3 --keep-ac3-stereo --add-audio commentary --hevc --nvenc --target 4000 for any TV show that gets encoded. 10-bit is enabled by default when using nvenc, and I won't go back to 8-bit. The quality improvement is massive and noticeable.

By the way - if you have a recent-vintage Nvidia card, you're going to drool the first time you do an HEVC encode.

[skj-dev]

Just adding a +1 here for 10bit HEVC via NVidia (Turing in my case). So far my experience is it's better quality at a smaller file size. 😎

[TehMaster]

Sigh.. I guess there is no way for not to redo my library then..... I suppose we all suffer from the same OCD with having the perfectly encoded library.

Can any of you recommend a good card specifically for this? I could leverage my 1080ti, but thats from my gaming rig, and i imagine i'll end up buying something for this. Would a 950 suffice, or something in the 10 series?

[lisamelton]

@TehMaster LOL! That's exactly the card I'm using. On my gaming rig, too! :)

Seriously, the 1080 Ti is actually and weirdly faster than the newer cards. I would use that.

[khaosx]

I'm using the 1660ti, though we've seen almost zero difference in speed with the non-Ti version. The main benefit to the Turing cards is the ability to turn on temporal AQ which may or may not give you better output depending on the input. @donmelton uses a 1080ti, and gets better speed than our Turing GPU's, and I think an additional nvenc instance.

It's worth noting that we also tested the RTX series, and it did not speed anything up.

[TehMaster]

Running --hevc --nvenc --target 3000 --eac3 i am averaging 240FPS and 10.3X speed using my 1080TI on my gaming rig... Silly fun.

Now im looking at ebay for a used 1080ti for my server... I blame you all for this. :)

[khaosx]

@TehMaster Quadro P2000 will rock your server's world, and requires very little power. Also, uncapped nvenc sessions.

[TehMaster]

@khaosx I had originally planned for a P2000, and then was told it wasnt as great for HVEC as other cards (reddit)... I think the P2000 would be perfect for transcoding and Plex use, and i'll take your word for it as well!

[wintervaler]

This could be pretty niche, but thought if it was relevant anywhere it might be this thread - does anyone have thoughts or experience on adding an Nvidia card to an Unraid server and running other-transcode in a virtual Windows machine, provided I get proper hardware pass-through working?

I know the GPU pass-through is possible in an Unraid Windows VM as people do it for gaming etc. -- but I ALSO know that Don says we should avoid trying to run other-transcode in virtualized environments so I don't want to risk dropping the cash for something that's going to end up being a fool's errand.

The alternative I guess is buying or building a completely separate cheap-ish Windows mid tower just to house the GPU and use it as my transcoding machine.

[klogg416]

@wintervaler I have the exact set up you described and it works brilliantly. My unRAID server has an iGPU in the Intel chip which is set as the primary graphics card in the UEFI, and I pass a GTX 1660 to a Windows VM. Read and write files over mapped network drives to the same server taking advantage of the virtio drivers.

My suspicion with Don's warning is it adds quite a lot of complexity, which then is a support nightmare. However if you know enough to be able to spin up and VM and pass a dedicated GPU to it, it will do exactly what you want, plus you get the added benefit of being on the Windows Nvidia driver path.

[klogg416]

One other advantage, you can share your GPU with Plex if you move Plex into that VM. Alternatively you can use @ttyS0 's docker container for other-transcode and the unofficial unRAID build that has Nvidia hardware support built in. But that is layers of unsupported and jenky and @donmelton is probably hitting me with the ban hammer as you read this. My more humble and sincere apologies @donmelton for bringing up this level of insanity in your support channel!

[wintervaler]

Wow. @klogg416 you have saved me so many hours researching and wondering. This is super helpful, thank you. I will report back with results!

As far as your Plex-related addendum, my actual Plex server (not the media, which is -- or soon will be -- on the Unraid NAS) is stored on a small HP 290 which has an 8th Gen Celeron with good quality QSV. It's slow when using other-transcode as I've described in another issue here, but does real-time transcoding for Plex very well if I ever need it. I'm just looking for something a bit beefier to crunch my blu-rays on a reasonable timetable (and to make sure I'm getting the best quality out of those transcodes).

[heelphan]

I can second @klogg416, I have the exact same setup - Unraid with a Windows 10 VM with my Nvidia 1650 Super passed through to it. Other-transcode is working beautifully. There are some great threads in the Unraid community and of course SpaceInvaderOne's great video tutorials on YouTube explaining how to get that VM with the video card passed through easily.

[n0rbertg]

Just wanted to add a +1 to the Unraid setup (@klogg416). I have a 2080 RTX passed through to a Win10 VM and a P2000 shared with multiple docker containers (using Linuxserver.io's NVIDIA driver)

As others have already mentioned other-transcode works well in Win10 (from this point of view it's basically a native box) but I can confirm ttyS0's docker container also works without a hitch from the Unraid shell. What I am thinking about but haven't tried yet is running it from inside a container - where other transcoding tasks are running - but this setup (docker inside a docker) might get messy. Even if it should work in theory.

[klogg416]

Fun update for @n0rbertg @heelphan and @wintervaler. Per the unraid 6.9 beta release notes:

"In a future release we will include the NVIDIA and AMD GPU drivers natively into Unraid OS. The primary use case is to facilitate accelerated transcoding in docker containers. For this we require Linux to detect and auto-install the appropriate driver. " Looks like taking the container approach for Plex and other-video-transcoding is about to get less shady in future releases.

full post here: https://forums.unraid.net/bug-reports/prereleases/unraid-os-version-690-beta22-available-r955/

[vasuvasu]

Tangentially related:

Are there any other flags that can be set to eek out more quality other than enabling --nvenc-temporal-aq? With video_transcoding I'd use handbrakecli's veryslow preset with x264 and slow with x265_10bit though I wanted to use the slower presets but couldn't afford to because less than 1fps wasn't working out so well for me 😂

Anything else to set to make an Nvidia card sweat more? What does other-transcoding's preset option do?

[lisamelton]

@vasuvasu Sorry I missed your question months ago. I think I answered you in another issue though.

Folks, I'm closing this issue now since I don't think there's any action to take at this point.