Transconde-video on windows failing to run intermittently

[eltito51]

Don - Sam, as per our exchange on Tweet I am loading the .log files of my transcoding session this morning on Windows10 (Latest version). Basically Transcode work with some files and halt or refuse to run with others. Those are the same exactly files I have successfully process under the same machine with "Linux Mint". Originally I have the following message:

"/local/bin/transcode-video:transcoding failed: Input/output error @ io_write -

As per Sam suggestion, this morning I try the --dry-run and regardless the file transcode-video halt after scanning the file (No message output) or just halt with the following message:

"/usr/local/bin/transcode-video: scanning failed"

I have a bunch of .MKV movies in a folder and to discard any issues with the "Path" i am copying one by one in an exclusive transcoding working folder within it I run my "Bash" command as per tghe installation instructions. Some files Transcode-video just run fine but some others I have all the above mess. This make me thing that the issue is not related to the installation or the path but most likely some weird issue with the metadada as Sam mention on tweet.

I am attaching the picture of the file folder order i am using to troubleshoot so far the only one that run without any hiccup is the "The Shawshank Redemption 1994" (Log is attached).

From the .MKV folder I try 6 movies and 2 were completed without issues. (Number 4-6). I repeat number 4 and Transcode halt after 80+%

The Shawshank Redemption 1994.log

The Belko Experiment 2016.log

Labyrinth 1986.log

I hope the information is helpful to fix the bug or If you guys believe I am doing something wrong your help is highly appreciated.

[eltito51]

Just a correction "Labyrinth 1986" Halt @ 80+%

[lisamelton]

@eltito51 Thanks for filing this, my friend! I'll let @samhutchins take lead on it since I'm still sick.

[eltito51]

No Problems Don, Anything that I can do to make Transcode better is a pleasure. Hopefully I was detailed enough.

Cheers

[samhutchins]

I'm not sure if you're having exactly the same problem that I had or not...

The error I sometimes get is /usr/local/bin/transcode-video: transcoding failed: Input/output error @ io_write - <STDOUT>, and I get it on "Alien³". Notice the superscript 3 there, I think that's what screws it up. If that's in either the filename or any of the metadata I get the failure, but if I remux to remove the "Movie Name" metadata it all works fine.

Could you install mediainfo (sudo apt-get install mediainfo) and paste the output of mediainfo for one of the movies that doesn't work? I wonder if there are other troublesome characters in there...

[eltito51]

let me work on that

[eltito51]

Unique ID :	260876923903018532081068155359239996256 (0xC4431165766874E82F2EAE932E2E0360)
Complete name :	D:\Movies_Downloads\The Girl On The Train 2016.mkv
Format :	Matroska
Format version :	Version 2
File size :	28.0 GiB
Duration :	1 h 52 min
Overall bit rate mode :	Variable
Overall bit rate :	35.8 Mb/s
Movie name :	The Girl on the Train
Encoded date :	UTC 2017-12-27 23:15:13
Writing application :	MakeMKV v1.10.7 linux(x64-release)
Writing library :	libmakemkv v1.10.7 (1.3.3/1.4.4) x86_64-linux-gnu

Here is the output from MediaInfo

[samhutchins]

Is that the full output?

I don't see any weird characters, so it looks like my hunch may be wrong. I'll keep digging

[klogg416]

Sam, I was getting the exact error you describe, only my cause was Japanese characters in one of the audio track titles. The job would fail out as it scanned the source file before transcoding started. Took me a while to discover it was related to the unknown characters, ultimately I ran it on a Linux/unRAID server and noticed the characters in the audio track name where I normally saw that error.

On Fri, Jan 19, 2018, at 4:40 PM, Sam Hutchins wrote:

Is that the full output?

I don't see any weird characters, so it looks like my hunch may be wrong. I'll keep digging> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub[1], or mute the thread[2].>

Links:

1. https://github.com/donmelton/video_transcoding/issues/189#issuecomment-359096662
2. https://github.com/notifications/unsubscribe-auth/AHAaMEDamPN0KCJyLPIy7BpswvRnICQyks5tMQuzgaJpZM4Rk5N_

[klogg416]

Came to the computer to provide more detail. One of the things that struck me as really odd is that the Japanese characters weren't visible in MKVtoolnix or mediainfo, however the audio tracks were coded as Japanese, so it seems that perhaps Handbrake fills in the characters based on the language encoding?

The thinking is I wonder if @eltito51 has other language tracks and maybe a surprise umlaut or something equally challenging. Unfortunately I didn't find a good solution, in the interest of time I just carved the audio and subtitles into a separate .mka file and left only the video track in the mkv, which transcoded without trouble. Then I pulled the unconverted audio and subtitles back into the transcoded MKV for a complete file. Not elegant, but functional.

Here is a clip of the output where it doesn't fail (unRAID), notice the characters: Metadata: filename : Profile-Regular.otf mimetype : application/x-truetype-font [02:40:04] scan: decoding previews for title 1 [02:40:05] scan: audio 0x1: flac, rate=48000Hz, bitrate=1 日本語 (FLAC) (2.0 ch) [02:40:05] scan: audio 0x2: ac3, rate=48000Hz, bitrate=256000 English (AC3) (2.0 ch) Scanning title 1 of 1, preview 9, 90.00 %[02:40:05] scan: 10 previews, 1920x1036, 23.976 fps, autocrop = 0/0/0/0, aspect 16:9, PAR 1:1 [02:40:05] libhb: scan thread found 1 valid title(s)

And here is the complete mediainfo dump, notice that the Japanese audio track (#1) doesn't have the name, only the language is set as Japanese: General Unique ID : 228111722025235074196861074835157395950 (0xAB9CB838741FEC615B750A129065A9EE) Complete name : Nausicaa Of The Valley Of The Wind (1984) Bluray-1080p FLAC.mkv Format : Matroska Format version : Version 4 / Version 2 File size : 8.68 GiB Duration : 1 h 57 min Overall bit rate mode : Variable Overall bit rate : 10.6 Mb/s Movie name : Nausicaa Of The Valley Of The Wind Encoded date : UTC 2018-01-14 19:12:01 Writing application : mkvmerge v19.0.0 ('Brave Captain') 64-bit Writing library : libebml v1.3.5 + libmatroska v1.4.8

Video ID : 1 Format : AVC Format/Info : Advanced Video Codec Format profile : High@L5 Format settings, CABAC : Yes Format settings, ReFrames : 7 frames Codec ID : V_MPEG4/ISO/AVC Duration : 1 h 57 min Bit rate : 9 143 kb/s Width : 1 920 pixels Height : 1 036 pixels Display aspect ratio : 1.85:1 Frame rate mode : Constant Frame rate : 23.976 (24000/1001) FPS Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 8 bits Scan type : Progressive Bits/(Pixel*Frame) : 0.192 Stream size : 7.47 GiB (86%) Writing library : x264 core 152 r2851 ba24899 Encoding settings : cabac=1 / ref=7 / deblock=1:0:0 / analyse=0x3:0x113 / me=umh / subme=11 / psy=1 / psy_rd=0.40:0.00 / mixed_ref=1 / me_range=32 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=3 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=230 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=2pass / mbtree=1 / bitrate=9143 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.40 / aq=2:0.60 Language : Japanese Default : Yes Forced : No

Audio #1 ID : 2 Format : FLAC Format/Info : Free Lossless Audio Codec Codec ID : A_FLAC Duration : 1 h 57 min Bit rate mode : Variable Bit rate : 1 180 kb/s Channel(s) : 2 channels Channel positions : Front: L R Sampling rate : 48.0 kHz Frame rate : 10.417 FPS (4608 spf) Bit depth : 24 bits Stream size : 988 MiB (11%) Title : Original [FLAC] Writing library : libFLAC 1.2.1 (UTC 2007-09-17) Language : Japanese Default : No Forced : No

Audio #2 ID : 3 Format : AC-3 Format/Info : Audio Coding 3 Format settings, Endianness : Big Codec ID : A_AC3 Duration : 1 h 57 min Bit rate mode : Constant Bit rate : 256 kb/s Channel(s) : 2 channels Channel positions : Front: L R Sampling rate : 48.0 kHz Frame rate : 31.250 FPS (1536 spf) Bit depth : 16 bits Compression mode : Lossy Delay relative to video : 7 ms Stream size : 214 MiB (2%) Title : English 2.0 Language : English Service kind : Complete Main Default : Yes Forced : No

Text #1 ID : 4 Format : ASS Codec ID : S_TEXT/ASS Codec ID/Info : Advanced Sub Station Alpha Duration : 1 h 55 min Bit rate : 59 b/s Count of elements : 1149 Compression mode : Lossless Stream size : 50.2 KiB (0%) Language : English Default : No Forced : No

Text #2 ID : 5 Format : UTF-8 Codec ID : S_TEXT/UTF8 Codec ID/Info : UTF-8 Plain Text Duration : 1 h 55 min Bit rate : 34 b/s Count of elements : 1164 Stream size : 29.2 KiB (0%) Title : English Language : English Default : No Forced : Yes

Menu 00:00:00.000 : en:00:00:00.000 00:01:59.995 : en:00:01:59.995 00:07:43.046 : en:00:07:43.046 00:12:10.104 : en:00:12:10.104 00:18:39.452 : en:00:18:39.452 00:23:34.747 : en:00:23:34.747 00:28:38.717 : en:00:28:38.717 00:33:30.175 : en:00:33:30.175 00:38:15.251 : en:00:38:15.251 00:44:57.111 : en:00:44:57.111 00:49:53.032 : en:00:49:53.032 00:54:40.611 : en:00:54:40.611 00:58:27.421 : en:00:58:27.421 01:01:19.426 : en:01:01:19.426 01:07:53.653 : en:01:07:53.653 01:13:51.177 : en:01:13:51.177 01:18:39.131 : en:01:18:39.131 01:24:27.896 : en:01:24:27.896 01:30:49.861 : en:01:30:49.861 01:34:39.340 : en:01:34:39.340 01:40:10.129 : en:01:40:10.129 01:46:06.860 : en:01:46:06.860 01:49:44.036 : en:01:49:44.036 01:52:08.930 : en:01:52:08.930 01:55:04.105 : en:01:55:04.105

[samhutchins]

@klogg416 I think the best solution for now is to either configure MakeMKV to only rip the audio tracks you want, or to remux with mkvtoolnix to remove everything you don’t want.

[eltito51]

Guys, I will keep digging on this on my side. Something that is really weird is that those same movies transcode without any issues under linux mint.

[lisamelton]

@eltito51 @samhutchins and @klogg416 My apologies for being AFK for so long. I'm feeling a little better today.

And my thanks, as always, to Sam for picking up the slack!

Reviewing the data so far, it's still unclear whether the problem is in El tito's Windows configuration, HandBrake's handling of his particular movies on Windows (and whether that's metadata-related), or some Windows-specific bug in my code.

But at least El tito doesn't have any problems using Linux Mint! That's both heartening and frustrating since that doesn't bring us much close to figuring out what's going on on Windows.

Still, I have faith that we will solve this. My only advice for El tito as he continues to dig is to reduce the failure to the simplest possible case. For example, does using HandBrakeCLI directly on one of these movies without transcode-video also fail? If that's the case then we may have a HandBrake bug here.

[samhutchins]

I've installed video_transcoding 'natively', which is to say without using the Bash on Ubuntu on Windows feature that was introduced in Windows 10. With that installation I don't have this problem, so I suspect the issue is unique to the Linux Subsystem for Windows. @eltito51, if you're interested in pursuing a 'native' installation you can see instructions on how to do so here (when installing Ruby, use Ruby 2.4 and uncheck the box in the last step. It's about ridk install and you don't need it for video_transcoding. There is also no longer a fonts directory in the HandBrake zip file, you don't need it. I'll update the instructions soon).

Further to that, back to the LXSS installation: If I call HandBrakeCLI on a troublesome file it works fine, so I'm gonna start hacking around in transcode-video to try and get to the bottom of what's going on. I'm not sure if there's a bug in there or not. My gut says no, the fact that this works on a proper Linux install suggests it's some quirk of this Linux on Windows thing.

[eltito51]

Guys, I will keep digging on this on my side. Something that is really weird is that those same movies transcode without any issues under linux mint.

[eltito51]

Sam:

The fact that some files transcode and some other fail make me agree with you that the issue should be with the Bash on Ubuntu on Windows. I will pursue the native installation as you suggest

[hilldw11]

[22:00:50] scan: audio 0x1: ac3, rate=48000Hz, bitrate=640000 English (AC3) (5.1 ch) (Dolby Digital EX)
[22:00:50] scan: audio 0x2: ac3, rate=48000Hz, bitrate=448000 English (AC3) (5.1 ch)
[22:00:50] scan: audio 0x3: ac3, rate=48000Hz, bitrate=640000 Francais (AC3) (5.1 ch)
[22:00:50] scan: audio 0x4: ac3, rate=48000Hz, bitrate=448000 espa/usr/local/bin/transcode-video: transcoding failed: Input/output error @ io_write - <STDOUT>

I think I am having a similar problem for one of my movies. For some reason I ripped it with additional audio tracks. It looks like the process is hanging up on the 4th track, which is Spanish. Is there a way to remove that one or do I just need to re-rip? thanks.

[lisamelton]

@hilldw11 Weird. It appears HandBrakeCLI is hanging during the scan.

There's no need to re-rip the video. You can remux it instead using mkvmerge.

If you give me the console output of mkvmerge -i video.mkv, I can write the more complicated mkvmerge command to exclude the Spanish language track for you.

[hilldw11]

File 'Videos/American_Sniper.mkv': container: Matroska Track ID 0: video (MPEG-4p10/AVC/h.264) Track ID 1: audio (AC-3/E-AC-3) Track ID 2: audio (AC-3/E-AC-3) Track ID 3: audio (AC-3/E-AC-3) Track ID 4: audio (AC-3/E-AC-3) Track ID 5: subtitles (HDMV PGS) Track ID 6: subtitles (HDMV PGS) Track ID 7: subtitles (HDMV PGS) Chapters: 13 entries

Hopefully this is correct.... thank you

[lisamelton]

@hilldw11 I also need to know which of those audio tracks is the Spanish one. I'm assuming that's the last audio track? That is, track ID 4?

[hilldw11]

@donmelton Yes, track 4 is spanish. I just need track 1 for english audio.

[lisamelton]

@hilldw11 OK, to leave off the last track then do something like this:

mkvmerge --output "American_Sniper.mkv" --audio-tracks 1,2,3 "Videos/American_Sniper.mkv"

To just select the English track then do something like this:

mkvmerge --output "American_Sniper.mkv" --audio-tracks 1 "Videos/American_Sniper.mkv"

[hilldw11]

@donmelton Thank you, its working now. I appreciate it.

perhaps i need to create a separate thread but just a quick question. How do you ensure that detect-crop comes out with the right values? Two different movies I was going to transcode today gave me odd advice, one was 134:130:0:0 and the other 22:22:4:2. Shouldn't they be symmetrical like this one is, 140:140:0:0 ? thanks again

[lisamelton]

@hilldw11 No problem. That's why I'm almost always online. :) I'm glad it's working.

As to your question about detect-crop. I can't ever ensure that it will produce the correct values because both HandBrakeCLI and ffmpeg fail sometimes at automatic crop detection. They're only right about 90-95% of the time. And while they're the best tools for the job, their analysis is still not good enough. That's why the default behavior for transcode-video is to not crop at all.

I always examine the output of detect-crop using the preview commands it prints out. And I make adjustments as necessary. I recommend you do the same.

[eltito51]

Sam, The native installation work as a Charm. I have a comment regarding your Instruction: "From the HandBrake zip file extract HandBrakeCLI.exe and the fonts\ directory to C:\bin" The Hanbrake CLI Zip file doesn't containt any "\Fonts Folder" just the executable file alone. DO WE REALLY NEED THAT FOLDER? SO FAR TRANSCODE SEEN TO WORK WITHOUT ANY ISSUES.

On the other side it look like the native installation is way faster than using the undelaying unbuntu bash system. I do imagine it make sense?

I am repeating some other movies to confirm everything work as expected.

[lisamelton]

@eltito51 Good to hear it's working now! Let us know if the other movies are working too and I think we can close this issue.

[samhutchins]

@eltito51 No, that Fonts folder is no longer required. I'll update the instructions this week

[eltito51]

@donmelton The other movies are transcoding perfectly. I am closing the ticket. @samhutchins Thanks everyone to help with this issue.

[sojojo]

@eltito51, I know this issue is closed, but I wanted to share an alternative solution to the foreign language audio/subtitle tracks issue on Ubuntu for Linux Windows Subsystem: use Debian instead. It does not have this problem.

[klogg416]

@sojojo, you got my hopes up! I installed Debain through the Windows store last night, followed Sam's set up guide for WSL, but encountered the exact same error on special characters when transcode-video scans the file. The title that it crashed out while displaying is "Bron. Säsong / Sæson 4 Episode 01 SE DK". It continues to work on a Native install and @ntodd's docker container on unRAID.

Out of curiosity, did you do anything beyond setting up the Debain flavour of WSL to increase its character support? Is your Windows region configured as USA?

Input #0, matroska,webm, from '../S04E01 - Episode 1 - Bluray-720p.mkv': Metadata: encoder : libebml v1.3.5 + libmatroska v1.4.8 creation_time : 2018-04-08T05:00:57.000000Z Duration: 00:57:32.04, start: 0.000000, bitrate: 7855 kb/s Chapter #0:0: start 0.000000, end 277.160000 Metadata: title : 00:00:00.000 Chapter #0:1: start 277.160000, end 579.840000 Metadata: title : 00:04:37.160 Chapter #0:2: start 579.840000, end 1201.600000 Metadata: title : 00:09:39.840 Chapter #0:3: start 1201.600000, end 1856.120000 Metadata: title : 00:20:01.600 Chapter #0:4: start 1856.120000, end 2451.520000 Metadata: title : 00:30:56.120 Chapter #0:5: start 2451.520000, end 3043.320000 Metadata: title : 00:40:51.520 Chapter #0:6: start 3043.320000, end 3452.040000 Metadata: title : 00:50:43.320 Stream #0:0(swe): Video: h264 (High), yuv420p(progressive), 1280x720, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 1k tbn, 50 tbc (default) Metadata: title : Bron. S/usr/local/bin/transcode-video: transcoding failed: Input/output error @ io_write -

[vickash]

I ran into this issue today on the latest Windows 10 and Ubuntu from the store. Seeing that the problem was to do with accented characters in subtitle language names, I first tried setting all subtitle languages to undefined in the source file metadata before transcoding, and that worked.

Then wanting to test if it was specifically displaying the characters that was the issue, I tried redirecting STDOUT and STDERR to a file, by appending &> video_log.txt to the transcode-video command. This worked and kept all the subtitle languages in tact.

Sharing this here in case anyone needs a workaround until the underlying issue gets fixed. You lose on-screen feedback, of course, but you can keep an eye on the log file that this gem natively generates. The file from STDOUT isn't very readable.

[lisamelton]

@vickash Thanks! I'm starting to suspect this might be an issue with parsing the text output of the initial scan from HandBrakeCLI to get the media information. Which makes me think this is yet another good reason to move to using the new --json API to get that data. Maybe it doesn't have a problem with special characters?

[vickash]

Yes, it looks like the parsing in Ruby modifies these characters somehow that makes them impossible to display? If I do HandBrakeCLI --scan on the same system with the same files, accented characters display correctly, and Japanese characters show as rectangles with questions marks, so I'm probably just missing fonts. In both cases, the scan finishes completely without crashing on the first one of these characters, which is the behavior of transcode-video.

[lisamelton]

@vickash Interesting. This seems like some really weird character encoding problem.

[joshstaiger]

Hey all, this issue has been bugging me for months. I finally got a chance to dig in, and I think I found the problem and have a fix.

See https://github.com/donmelton/video_transcoding/pull/263 for details.

[lisamelton]

@joshstaiger Awesome detective work, sir!

[lisamelton]

@joshstaiger Re-opening this because of your fix. Sorry, I forgot to do that yesterday.

[lisamelton]

@joshstaiger OK, I just took your second patch, #264. I'm going to leave this issue open until the next release and until you, @samhutchins and I can review whether any other code using HandBrakeCLI or any other tools needs the same patch.

[lisamelton]

@joshstaiger @samhutchins After reviewing all the code tonight, I think the only area we need to look at in depth is the call to ffmpeg in convert-video at line 434 in that file. Why? Because that's the only call to IO.popen which processes output to the console one character at a time. All other calls process output from various other tools line by line, which should not be problematic.

Josh, have you seen the same kind of bug using convert-video?

[joshstaiger]

To be honest, I've never used convert-video before. I'll give it a go on a file that has exhibited the issue when I get home a little later.

[lisamelton]

@joshstaiger BTW, your buffering patch is so straightforward, that I could see applying it for the call to ffmpeg in convert-video anyway just as a preventative measure on Windows. //cc @samhutchins

[joshstaiger]

Okay, I ran convert-video over a few of the files in question, and at first blush I don't see the problem.

The primary offenders for me have always been language names for audio or subtitle tracks ("español"), and it seems that ffmpeg only prints the iso abbeviations, eg: Stream #0:5(spa), rather than the full language name.

However, I suppose ffmpeg could conceivably print other metadata with multi-byte characters. Maybe localized track names? — I'm not sure.

And in fact, I noticed that ffmpeg prints out the source filename before the conversion. And I can trigger the crash if I I rename my input file to "ñ.mkv" or "語.mkv". And in both cases a similar buffer patch to convert-video also fixes the problem.

[lisamelton]

@joshstaiger Very thorough detective work, sir! Thank you so much!

Well, that settles it. I'll apply your patch to the ffmpeg code tonight then.

[lisamelton]

@joshstaiger OK, here's the patch:

diff --git a/bin/convert-video b/bin/convert-video
index 2f7aba6..5d21528 100755
--- a/bin/convert-video
+++ b/bin/convert-video
@@ -436,8 +436,23 @@ HERE
             Process.kill 'INT', io.pid
           end

+          buffer = ''
+
           io.each_char do |char|
-            print char
+            if (char.bytes[0] & 0x80) != 0
+              buffer << char
+            else
+              if not buffer.empty?
+                print buffer
+                buffer = ''
+              end
+
+              print char
+            end
+          end
+
+          if not buffer.empty?
+            print buffer
           end
         end
       rescue SystemCallError => e

Let me know if that looks right. And I'm testing it now.

[joshstaiger]

Yep, that looks right!

[lisamelton]

@joshstaiger And my testing did turn up any problems so I just checked it in. :) Thanks for quick review and coming up with this fix in the first place!

I'll leave this issue open until the release goes out next week.

[joshstaiger]

One other thing I'll note for completeness is that the crash seems to be limited to wsl (Windows Subsystem for Linux), which probably explains why it was so hard to track down in the first place.

I installed a plain Windows Ruby and video_transcoding setup and it looks like without the buffer patch it merely mangles the multi-byte output ("espa��ol") without crashing,

... and the patch seems to fix the mangling, too, if the Windows Console codepage is set to UTF-8.

[lisamelton]

@joshstaiger Interesting. So, win-win, then! :) Thank you, again. You will be mentioned in the Release Notes, sir.

[lisamelton]

@joshstaiger OK, this has now been released in version 0.25.0 of the Gem. Thanks again for all your help!