Performance slows massively over time

[nickrupert7]

Hi Don, Thanks for your help on my last issue! I got a script working to detect English foreign audio subtitles and add it to the transcode-video parameters. However, now I'm having a new issue that I've never seen before, and while it may or may not be more about HandBrakeCLI than transcode-video, you really seem to know your stuff with this, and I was hoping you might have some insight.

Before I started using transcode-video a few weeks ago, I was using another custom written script to run handbrake on my MKVs with absolutely no performance issues (used to copy 100+ movies for 1 year+). Now all of a sudden, Im seeing a MAJOR drop in performance, even with the --quick option. transcode-video will start running on my MKV blu-ray rip with an ETA of about 3 hours, which is already a bit slower than I'm used to, but as I watch it over time, the ETA more than doubles to about 6 1/2 hours after it's been running for over an hour! I tried it on another file and it was even worse - predicting 12 hours to run. It's still running now, but I'll post the log "as-is" so you can see what it looks like to this point.

Earlier, I ran this exact script on a DVD and it completed the ripping and compressing in about an hour. I know Blu-Rays are a bit more complicated and will take more time, but I'm shocked that it grows to 6.5 hours, which tells me something's gone wrong.

Some other info you might need to know: My server that this runs on is Ubuntu 17.04. In that server, I'm sporting an Intel Celeron 2.7Ghz dual core processor, so not the beefiest machine in the world, but again, I've never had performance issues like this before. The DVD that worked earlier was Star Wars: A New Hope, and my script that rips, detects forced subtitles, and compresses using transcode-video ran in almost exactly an hour. The Blu-Ray I'm trying now is Marvel's The Avengers, which ripped just fine, but now the compression time is lagging to no end. I also noticed in the log that initially the frame rate was about 16 fps, but now it's degraded to about 7 fps.

Thanks in advance! You're the man!

transcode-video.log.txt

[nickrupert7]

Update: Somehow, 30 minutes after I made this post, the transcoding finished?? Perhaps it's a bug in my script that detects when it's finished because the output of transcode-video which I pipe to a log file still shows it as encoding... Very very weird stuff here

Update: I checked, and it appears that my script is detecting that transcode-video is done before it actually is. Luckily, it continues to run in the background, so that particular issue is probably something on my end. However, the performance is still an issue.

[lisamelton]

@nickrupert7 Sorry I took so long to respond. I was playing chauffeur this morning for my family.

Once transcode-video hands the task of doing the actual transcoding off to HandBrakeCLI, the only thing it really does is pipe the the console output to the .log file. So the impact from transcode-video itself on performance is minimal.

However, HandBrakeCLI and the x264 library it contains will both attempt to use every bit of your CPU to transcode your input. Which is usually what you want.

I don't have enough information from that .log file you attached to diagnose the slowdown, but my guess would be either that 1) you have other processes running on your server that are sucking down CPU cycles or 2) you're reading and/or writing input/output over the network and there's a slowdown on your network or the devices serving your files.

Are either of those scenarios possible?

[nickrupert7]

No no! By all means, I consider your reply to be speedy, considering other forums on GitHub. I've actually been looking in to it, and that update I posted actually set some alarms off in my head that could explain what's going on. As I mentioned, apparently my script is moving on from the transcode-video command before it's actually done (perhaps there's some kind of default time-out for a bash script?), but the log reveals that moving on doesn't actually kill transcode-video, so perhaps the instances are just piling up, and they're all running in the background. I've been working on it all this afternoon, and once I have results I'll let you know.

[lisamelton]

@nickrupert7 Ah, I see what you mean now. Yeah, instances could be piling up. You've got me curious now so I will await your findings.

[nickrupert7]

I restarted the computer to start with a fresh slate, and it's still giving an ETA of 6 hours, but it's a possibility that there are some setting differences from how I used to run it, combined with the fact that I'm not encoding subtitles in addition to the fact that it's a blu-ray instead of a DVD. 🤷‍♂️ I'll test it with a plain DVD tomorrow and see if it goes back to the speeds I'm used to, or if it's still suddenly acting slow.

[lisamelton]

@nickrupert7 Thanks for the update! I will await the results of your next test.

[nickrupert7]

I tested with my DVD copy of Return of the Jedi, and it went back to the 1 hour start-to-finish rip+transcode, so it really seem like it's the Blu-Rays that go slow, and there's not much to be done about it. Having said that, I added an exit trap and a startup condition to my script that will make it impossible for multiple instances to run at the same time, because I do believe that the bump to 12 hours was a result of instances piling up. Since I added those conditions, I haven't seen anything dramatic like 12 hours again, which is good. The other thing worth noting is that I compared the HandBrake ETA shown in the log to the ACTUAL runtime, and it's off by like two hours, meaning that the actual runtime was 4 hours, which isn't too bad at all.

All in all: ETAs predict longer than it actually takes, Blu-Rays take a lot longer than DVDs, and instances piling up could be a concern, so to anybody in the future that has this problem, watch out for the process invisibly running in the background.

Thanks for your help and enthusiasm, Don!

[lisamelton]

@nickrupert7 You are very welcome, sir! I'm glad you figured it out. Your modification to your script is a very good idea.

And you're right, transcoding Blu-ray Discs takes much, much longer than transcoding DVDs. Using the --quick option on my iMac, I typically get between 40-70 FPS with Blu-rays and 300-325 FPS with DVDs.