Chapter Image Extraction Partial Failure

[gcw07]

This feature works great for some of my movies and TV shows, but sometimes it seems to fail and only do part of the chapters within the movie/show. So for instance, say there are 23 chapters on a movie. Sometimes it will correctly do all 23, but others, it just does the first 2 or 3 chapter points.

https://docs.google.com/file/d/0B0pNvArEEZ2SOU1yRV9GVzBTbWM/edit?usp=sharing

Rerunning the chapter extraction doesn't seem to update it.

[tikuf]

This is a good indication that the seek index in the video is broken or damaged. To test try seeking past the point that the chapters fail to extract in wmp.

[gcw07]

WMP won't open a mkv, so not really an option. But seeks fine in MPC-HC.

[gcw07]

Is there any other method of testing?

[gcw07]

OK, tried it out on my HTPC (which is the only computer I have setup to play MKV's in WMP and isn't not the server), tested out the files it errored on. No problems seeking past the points. So don't think it is that problem.

[tikuf]

@gcw07 Could you run the following for me. Shut down mb3 delete C:\ProgramData\MediaBrowser-Server\data\extracted-video-images folder delete C:\ProgramData\MediaBrowser-Server\data\library.db Start mb 3 up let it rebuild the library.db once that has completed shut down mb3 again Start the server and run the extract chapter images task and then post a link to the server log (if the issue isn't fixed)

Thanks

[LukePulverenti]

@tikuf One thing you had me do is stop attempting to extract chapter images for a file as soon as one fails.

Is it possible that say, chapter 3 could fail and then the rest succeed?

[tikuf]

Generally not if we cant seek to chapter 3 (and seek would be the only real point of failure) then we should not be able to seek to chapter 4. If killing off the existing chapter images & db doesn't fix it I will give him a batch to try manually outside mb3.

[LukePulverenti]

I don't think it will because I have the same issue with some of my rips.

On Tue, Apr 23, 2013 at 11:54 AM, Simon notifications@github.com wrote:

Generally not if we cant seek to chapter 3 (and seek would be the only real point of failure) then we should not be able to seek to chapter 4. If killing off the existing chapter images & db doesn't fix it I will give him a batch to try manually outside mb3.

— Reply to this email directly or view it on GitHubhttps://github.com/MediaBrowser/MediaBrowser/issues/190#issuecomment-16867222 .

Luke Pulverenti

[tikuf]

Have you tried a manual extraction past the failure point? I wondering if a fast and slow seek my fix it? I cant repo here I am getting chapter images for everything I have looked at thus far.

[LukePulverenti]

No, I haven't. Try it with gcw07 and let me know what happens.

On Tue, Apr 23, 2013 at 12:08 PM, Simon notifications@github.com wrote:

Have you tried a manual extraction past the failure point? I wondering if a fast and slow seek my fix it? I cant repo here I am getting chapter images for everything I have looked at thus far.

— Reply to this email directly or view it on GitHubhttps://github.com/MediaBrowser/MediaBrowser/issues/190#issuecomment-16868145 .

Luke Pulverenti

[gcw07]

I did see this and I will try what tikuf suggested and post the logs. Just been busy so it will have to be a little later today.

[gcw07]

@tikuf is it supposed to rebuild the library.db right away? It hasn't recreated that file after 20 minutes.

[tikuf]

The server should start immediately rebuilding on start up. This is not happening?

[gcw07]

No it wasn't. It is a permissions problem that popped up earlier with updating the server software.

http://community.mediabrowser.tv/permalinks/r/13546/55493/mb3-server-beta#55493

I restarted the server as an administrator and it finally created the library.db file. I'll get a log on the other problem soon now.

[gcw07]

OK, it still failed. Here is the log. Lots of errors reading the files it seems, yet no problems seeking on them.

Logfile

[tikuf]

There are a lot just not as many as you may think (we pretty much get 2 errors for each item) still not good but not as bad as it looks from the logs.

Firstly we need to find out if the failed files have anything in common (video codec for example). If you could choose some of the failures and look at them in the library explorer see if you can find common ground.

If you could test the extraction manually for me with the two command lines below.

ffmpeg -ss 600 -i file:"input.mkv" -threads 0 -vframes 1 -filter:v select='eq(pict_type,I)' -vf "scale=iw*sar:ih, scale=600:-1" -f image2 -y "testimage.jpg"
ffmpeg -ss 600 -i file:"input.mkv"  -threads 0 -vframes 1 -vf "scale=iw*sar:ih, scale=600:-1" -f image2 -y "testimage.jpg"

In the code if the first c/l fails we revert to the second so if you could mimic the server in this aspect .

Open a cmd prompt in the server program data ffmpeg folder (same level as ffmpeg.exe) and test a few of your failed files incrementing the -ss 600 by 600 seconds each run (eg -ss 600 -ss 1200 -ss 1800 etc etc) until you are past the fail point. The "testimage.jpg" will be overwritten each time so check the image in between runs. Make sure you use the full path to the files you test when replacing "input.mkv"

If you see an ffmpeg failure can you tell me exactly what it says (or scr shot will suffice).

Thanks

[gcw07]

I don't see anything as far as codec that is different from ones that work. Basically they are h264 videos. I have others of the same codec that extracted fine and then those that didn't. All the failures do look like h264 videos though. I haven't come across one that is anything else yet. I'll work on the manual extraction and report back.

[gcw07]

Getting the message "Output file is empty, nothing was encoded (check -ss / -t /frames parameters if used)". Even if I reduce the seconds down it still says that.

[gcw07]

OK, tried reducing it to 0 and then jumping up by 60 seconds instead. 0 created a blank image like it should. 60 created an image, but also had some errors:

[h264 @ 03ff4400] reference picture missing during reorder [h264 @ 03ff4400] Missing reference picture, default is 65556 [h264 @ 03ff4400] reference picture missing during reorder [h264 @ 03ff4400] Missing reference picture, default is 65556 [h264 @ 03ff4400] reference picture missing during reorder [h264 @ 03ff4400] Missing reference picture, default is 65556 [h264 @ 03ff4400] reference picture missing during reorder [h264 @ 03ff4400] Missing reference picture, default is 65556 frame= 1 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A dup=0 drop=8 video:21kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.101875%

[ebr11]

gc - be sure the user under which MB server runs has full control of your mediabrowser-server folder. That should eliminate the need to run it as admin.

[tikuf]

Ok on that same file try running

ffmpeg.exe -i file:"input.mkv" -c copy "input-new.mkv"

exchanging the input.mkv for the path&filename to the file use same path for output (input-new.mkv) . Then move your original file somewhere outside mb3 and rename input-new.mkv to what ever the old input was (in other words swap out the file).

Run a refresh then run extract the chapter images. I am 99% sure it is a damaged index this doesn't always show up during playback as players are more tolerant than encoders.

I broke a heap of indexes today for testing and I was seeing exactly what you are seeing.

[gcw07]

That did not copy the entire file. It only copied the first 2 minutes of a 52 minute long video.

[gcw07]

@ebr11 it does appear the user has the permissions set, but shouldn't the installer/updater reset those permissions just in case any are wrong? And no idea if an updater can even reset permissions like that.

[ebr11]

The installer does attempt to set the permissions but, apparently, it doesn't always work.

[tikuf]

"It only copied the first 2 minutes of a 52 minute long video." Ok that tells me that the file is broken (for ffmpeg anyhow) Basically means it needs a full re-encode (or attempt at one to try and fix its issues).

If you throw me a link to a text file containing the library explorer data I will write one for you to run.

[gcw07]

https://docs.google.com/file/d/0B0pNvArEEZ2SZFpXS1dxUF9kYkk/edit?usp=sharing

Different file from the 52 minute video, but same problem.

Re-encoding 30 to 40 files is probably not in the cards. But good to know what is is.

[LukePulverenti]

@tikuf, this happens to me on some blurays ripped by me personally using MakeMkv. Are you saying there's nothing we can do?

[tikuf]

It usually happens when the time stamps skip backwards or something similar. If you try to skip to one of the chapters that doesn't have an image chances are it wont play (if you transcode from the start it will work as long as the file isn't superbad). For the most part

ffmpeg.exe -i file:"input.mkv" -c copy "input-new.mkv"

fixed mine (but i broke them on purpose so ymmv). A slow seek may work too (I dont know). Because we are not logging the actual ffmpeg stderr I dont know what is happening to those files (but if we did that we would have 40 trillian ffmpeg txt files) Can you please try the -c copy c/l on one of your failing files and see if it fixes it?

[gcw07]

@LukePulverenti that is what I also use to rip my files and all the "bad" ones appear to be blu-ray rips from MakeMKV. Re-ripping the bad movies wouldn't be that horrible for me, but re-ripping the TV series is the time consumer as most of the broken ones are entire seasons of episodes.

[tikuf]

You don't need to re rip. Just re-encode (hopefully) now i know that you both are using the same program I should be able to repo. Tell me are the ones you are having issue with 1:1 rips?

[LukePulverenti]

I don't want to re-encode. I'd rather just find a new ripping program if that's the case. My process is generally that I choose the highest quality audio track, usually omitting the core track, and then forced english subs only. For foreign language content I take the whole english subtitle track.

On Thu, Apr 25, 2013 at 12:23 PM, Simon notifications@github.com wrote:

You don't need to re rip. Just re-encode (hopefully) now i know that you both are using the same program I should be able to repo. Tell me are the ones you are having issue with 1:1 rips?

— Reply to this email directly or view it on GitHubhttps://github.com/MediaBrowser/MediaBrowser/issues/190#issuecomment-17017738 .

Luke Pulverenti

[gcw07]

I'm not sure what 1:1 rips are. I usually just go in, select the highest quality audio track and sub track (eg. DTS-HD and DTS; or DD-MA and DD), remove all subtitles and go.

[gcw07]

OK, so I re-ripped The Hobbit as that was one of my files that was broken. While it now gets a little further along (10 minutes in instead of the 5 it was failing at), it still doesn't work. Not sure what to do, but don't think re-ripping will fix it.

[tikuf]

It is the was makemkv does it it just joins the files together and the time stamps are all over the shop

[tikuf]

Gcw this command line should fix your makemkv files so they are not broken for ffmpeg

ffmpeg -fflags igndts -i file:"Input.mkv" -threads 0 -c copy "output.mkv"

Same instructions as above just a slightly diff c/l it will take about 8-10 mins per full bd . Hopefully it works for you i created a few files with makemkv today and it fixed all of them. It may desync the audio so dont blanket run it yet. if it produces thumbnails I will play with the sync.

[gcw07]

Just tried it. It only created about a 4 minute long video (from a 3 hour movie). Basically I think it stops at the same point the thumbnail stops working.

[tikuf]

Are you trying it on the same video each time? If so please try a different one (that has the same issue with thumbs) that one appears hosed.

[tikuf]

You could also try

ffmpeg -fflags ignidx -i file:"Input.mkv" -threads 0 -c copy "output.mkv"

which should ignore all indexes

[gcw07]

Tried it on a different file. It stopped early too (2 minute mark of a 52 minute file). Tried the other command and it did the same. Just ends early.

[tikuf]

Ok these are tv files they may be a lost cause it looks like makemkv just slices them up without any thought to the consequences (I could be wrong). Please try on a movie that you have the same issue with.

How big are your tv files? I wouldn't mind looking at one to see if i can sort it (these question may be asked by other people) I would like to have a better idea of what is going on. (if you have a short one that would be great).

[gcw07]

I've tried it on movie files and tv files. Does the same exact thing for both. At the point the thumbnails stop producing, the copy stops. I've even tried re-ripping a movie and trying it on that. It fails too.

The smallest one I have is 4.5 GB. Not sure how I could transfer something that large to have you check out and test.

[ebr11]

it will take about 8-10 mins per full bd

Holy crap. That's 83 hours for 500 movies. Doesn't sound like a process we want to undertake...

[LukePulverenti]

TIkuf, are you saying this is flat out not possible to do on MakeMKV rips? It's ok if the answer is yes I just want to get an accurate picture.

FWIW, I believe it works perfectly on rips that are sourced from a single m2ts.

On Fri, Apr 26, 2013 at 9:31 AM, Eric Reed notifications@github.com wrote:

it will take about 8-10 mins per full bd

Holy crap. That's 83 hours for 500 movies. Doesn't sound like a process we want to undertake...

— Reply to this email directly or view it on GitHubhttps://github.com/MediaBrowser/MediaBrowser/issues/190#issuecomment-17074416 .

Luke Pulverenti

[tikuf]

Well never say never but it looks like it the way the mkemkv cuts and splices the video (older versions of makemkv at anyrate). I have 1.81 that i d/l today and am running in trial created video that was not seekable and the first command I posted fixed the files so it was seekable by ffmpeg. I did notice in their change log for this version "Improved compatibility of produced MKV files". I will try to get an older version and see what happens. In the meantime it is worth trying the first c/l on one of your movies ( I don't think we can properly decode a tv rip). Unless there is a way for ffmpeg to seek using pts rather than dts (there has been some patches submitted but no commit)

@ Eric yes it is a considerable time it could be staged but offering a user a possible solution may be nice

[MrLinford]

Is only MKV that has the issue gents?

I know myself and others that would find captures and seeking really useful. I'll do some digging when I start testing later/weekend.

Craig.

Sent using the force!

-----Original Message----- From: Simon notifications@github.com Date: Fri, 26 Apr 2013 07:05:13 To: MediaBrowser/MediaBrowserMediaBrowser@noreply.github.com Reply-To: MediaBrowser/MediaBrowser reply@reply.github.com Subject: Re: [MediaBrowser] Chapter Image Extraction Partial Failure (#190)

Well never say never but it looks like it the way the mkemkv cuts and splices the video (older versions of makemkv at anyrate). I have 1.81 that i d/l today and am running in trial created video that was not seekable and the first command I posted fixed the files so it was seekable by ffmpeg. I did notice in their change log for this version "Improved compatibility of produced MKV files". I will try to get an older version and see what happens. In the meantime it is worth trying the first c/l on one of your movies ( I don't think we can properly decode a tv rip).

@ Eric yes it is a considerable time it could be staged but offering a user a possible solution may be nice

---

Reply to this email directly or view it on GitHub: https://github.com/MediaBrowser/MediaBrowser/issues/190#issuecomment-17076167

[LukePulverenti]

I don't think it has anything to do with the container. It's the format of the video stream contained within.

On Fri, Apr 26, 2013 at 10:10 AM, Craig linford notifications@github.comwrote:

Is only MKV that has the issue gents?

I know myself and others that would find captures and seeking really useful. I'll do some digging when I start testing later/weekend.

Craig.

Sent using the force!

-----Original Message----- From: Simon notifications@github.com Date: Fri, 26 Apr 2013 07:05:13 To: MediaBrowser/MediaBrowserMediaBrowser@noreply.github.com Reply-To: MediaBrowser/MediaBrowser reply@reply.github.com Subject: Re: [MediaBrowser] Chapter Image Extraction Partial Failure (#190)

Well never say never but it looks like it the way the mkemkv cuts and splices the video (older versions of makemkv at anyrate). I have 1.81 that i d/l today and am running in trial created video that was not seekable and the first command I posted fixed the files so it was seekable by ffmpeg. I did notice in their change log for this version "Improved compatibility of produced MKV files". I will try to get an older version and see what happens. In the meantime it is worth trying the first c/l on one of your movies ( I don't think we can properly decode a tv rip).

@ Eric yes it is a considerable time it could be staged but offering a user a possible solution may be nice

---

Reply to this email directly or view it on GitHub:

https://github.com/MediaBrowser/MediaBrowser/issues/190#issuecomment-17076167

— Reply to this email directly or view it on GitHubhttps://github.com/MediaBrowser/MediaBrowser/issues/190#issuecomment-17076464 .

Luke Pulverenti

[tikuf]

Correct Luke but more the way its put together.

[gcw07]

@tikuf I am using 1.81 and tried re-ripping a movie and trying the command on it and it still stopped at a few minutes into the file. Just sounds like MakeMKV and this feature aren't going to be too compatible. Lot of work trying to figure this out when %40 of them are blacked out images anyways. Thanks for looking into it and trying.

[ebr11]

Perhaps we can talk to the MakeMKV guys and get them to fix it...

[LukePulverenti]

Tikuf can you report the issue to them? On Apr 26, 2013 11:52 AM, "Eric Reed" notifications@github.com wrote:

Perhaps we can talk to the MakeMKV guys and get them to fix it...

— Reply to this email directly or view it on GitHubhttps://github.com/MediaBrowser/MediaBrowser/issues/190#issuecomment-17082820 .