Closed Brandonshire closed 8 years ago
I do not encounter this issue, so may we use the same EyeTV Preferences to try to isolate the cause? Please tell EyeTV to export all recordings to iPad format under EyeTV>Preferences>iPhone>More options…
Record a test show. Do you see the issue?
By the way, my next request will be for you to babysit the recording/export/comskip process by watching Activity Monitor to see if these three steps happen sequentially. The comskip
process should kick in with 200% CPU cycles about a minute _after_ EyeTV is done with its iPad export. Is this what you see?
My preferences are, and already were, set that way (aside from the "require a passcode" option).
To be clear before I do this, do you want me to set up a smart guide, with the "export to iTunes" option enabled for this test recording? That's how i have my others set up. In the past whenever I choose the "export to iTunes" option manually I end up with commercials cut out, but when I have them automatically exported as soon as the show is over I have not.
I also want to add another piece of data: the recordings, within EyeTV all have the correct time on them. For example the episode of Nashville that I found the problem with last night is 1hr and 2 minutes within EyeTV (I always have it record one minute early and one after to account for slight differences in broadcast times) but the exported copy into iTunes is only 40minutes due to the commercials being cut out.
Also you say "By the way, my next request will be for you to babysit the recording/export/comskip process by watching Activity Monitor to see if these three steps happen sequentially." But then only mention one step. What are the others I should be looking for?
Hypothesis: You are exporting to iTunes with a different (e.g. iPhone, Apple TV) format than the iPad h.264 format that EyeTV creates with every recording. If so, this would cause EyeTV to transcode the show to the new format, and if commercials are marked they will be edited from the export.
If you export to iTunes with the same iPad format, there is no new transcoding step, and EyeTV simply copies the .iPad.m4v
file from the .eyetv
directory to iTunes.
Please describe the format you have configured for iTunes outputs. To avoid a new transcoding step that edits out commercials, you should have EyeTV>Preferences>General>Export> iPad/Apple TV button set to iPad.
Ok, it is in fact set to "AppleTV HD 1080P."
That certainly could b the problem, though in the past when I was using the old ComSkipper set up (using WINE etc.) this was (usually) not a problem. If i set it to iPad, what kind of resolution will I get?
What are the others I should be looking for?
The correct process looks like this, in this order:
comskip
(heavy comskip
process CPU usage)Ok, it is in fact set to "AppleTV HD 1080P."
That certainly could b the problem, though in the past when I was using the old ComSkipper set up (using WINE etc.) this was (usually) not a problem. If i set it to iPad, what kind of resolution will I get?
iPad is 720p h.264.
But there should be a way to get this to work. Hypothesis: In the case of a new transcoding for iTunes exports, RecordingDone and/or ExportDone are looking for the wrong processes to finish before triggering comskip
.
Would you please do these, using Activity Monitor:
comskip
runs before the Apple TV transcoding/export is completed.The two processes that RecordingDone looks for to complete are EyeTV
and Elgato H.264 Decoder
.
Ok, so I just recorded something new, set up a smart-guide with exporting to the AppleTV (and have left the AppleTVHD1080p setting as it is for now). The recording just finished and the relevant processes I'm seeing right now are EyeTV and Elgato MPEG-2 Decoder, EyeTV is running at about 275% and Elgato MPEG-2 Decoder is running at about 23 to 24%. I'll update once this part is done.
Ok, just after I said that comskip just kicked off at about 105%. Then it stopped. It looks like it only ran for a minute or two. The other two are still running at roughly he same percentages.
Ok, it finished exporting, and Elgato MPEG-2 Decoder dropped off, and EyeTV dropped down to between 7 and 10 percent. Oddly enough, despite being instructed to by the smart guide did not in fact export the recording to iTunes. I have another test recording going now with a smart-guide I set up before the recording started I'll try to watch what happens there after I'm done.
I will rely upon you to tell me:
It will be helpful in watching this to right-click on the EyeTV show>Show in Finder so you can see when the Apple TV .m4v is created and when the iPad .m4v is created, and your assessment of which processes produced them.
The logic here is to figure out which processes to wait to complete before triggering comskip.
This would all be a lot simpler if Elgato had a "ExportStarted" script, but they don't.
Ok, so a recording just finished exporting. I don't have exact times but here's what happened.
First, once the recording was done and the exporting started the process EyeTV ramped up to around 200 to 275% processor usage (it varied of up and down throughout the process). Then Elgato MPEG-2 Decoder started up at around 7ish percent. Those both ran for several minutes, at which point ComSkip showed up at around 7ish percent for a minute or two and then dropped off.
Once the first export was done, a second one started. That one followed the same pattern as above, but I never saw ComSkip show up during that export. Once that export was done a copy of the episode showed up in iTunes.
The copy in iTunes has the commercials cut out of it. When I look at the episode in EyeTV they are not cut out though I can see the sections where they are marked on the scrubbing/progress bar for the episode.
Please tell me:
When a recording finishes:
1) EyeTV - Recording a show, roughly 10 to 15% 2) EyeTV - Seems to be the start of the exporting, roughly 200 to 275% 3) Elgato MPEG-2 Decoder - Seems to also be part of the above, roughly 7 to 10% shows up just after EyeTV ramps up to 200ish % 4) ComSkip (this shows up while the other two are running, for a couple of minutes and then stops, the other two continue running). I presume this is marking the commercials though I don't see any file produced.
Then after one round of that, if I look at the "Recordings" tab of EyeTV I see that the "Exporting" progress bar completes and then goes back to empty, and starts filling up again. In Activity Monitor I see:
1) EyeTV - Seems to be the start of the exporting, roughly 200 to 275% 2) Elgato MPEG-2 Decoder - Seems to also be part of the above, roughly 7 to 10% shows up just after EyeTV ramps up to 200ish %
I can see the file size of the recording for the episode increasing in size as this second round of "Exporting" After it is complete there is a copy of the episode in my iTunes Library. This copy has the commercials cut out.
If i manually tell it to export to iTunes again (click on the AppleTV HD 1080p button), the same thing as the "second round" export happens, an takes just as long, though the file size shown within EyeTV (or when I look at the file using "Show in Finder" doesn't go up.
Thanks, that's enough for me to go on. I'll make a few changes and post back after I cut a new release.
Now here is something interesting:
If I switch EyeTV>Preferences>General>Export to iPad, it will export a copy to iTunes quickly that is the correct length (it won't have commercials cut out). However it also doesn't seem to mark the commercials as chapter markers. The chapter markers in the episode are simply at roughly 5 minute increments and don't connect to the commercials, at all. Again, when I look at the recording within EyeTV, I can see where it's marked commercials and it's pretty accurate, and those sections have nothing to do with the chapters that I see in the "iPad" exported copy in iTunes.
Yes, I saw that bug and your posting this issue led me to the solution, so thanks for that.
Just like we must wait for exports to be done before running comskip, we must wait for comskip to be done before adding its commercials to iTunes exports.
I'll fix and post back.
Glad I could be of help! Thank you so much for all of your hard work on this!
@Brandonshire, I'm trying to code a more robust method of file-based signaling between scripts.
Would you please do me a favor and cd to the .eyetv
directory of a test recording during the recording/export process, and posting the contents using the command ls -l
during the four major steps:
I'll assume you know your way around the command line. Please ask if you have any questions. See the previous closed issue for examples.
Ok, doing this now. To be clear, do you want me to go to the directory the recording is IN, or the directory that is the .eyetv file (since as I understand they are actually basically just folders)? I'm trying to gather both sets of data right now, but of course only want to post what you actually want.
On the assumption that you want the stuff inside the .eyetv archive (which seems to make the most sense), here is what I found. Hope it's helpful! Let me know if you actually want the contents of folder containing the .eyetv file.
Recording:
total 5638552 -rw-r--r-- 1 stephenb staff 4694 Nov 13 07:58 000000001bf61603.eyetvp -rw-r--r-- 1 stephenb staff 256480 Nov 22 20:25 000000001c029c54.eyetvi -rw-r--r-- 1 stephenb staff 2446 Nov 22 19:59 000000001c029c54.eyetvr -rw-r--r-- 1 stephenb staff 2886664612 Nov 22 20:25 000000001c029c54.mpg
iPad Export:
total 6980056 -rw-r--r-- 1 stephenb staff 4330 Nov 22 20:32 000000001bf61603.eyetvp -rw-r--r-- 1 stephenb staff 316160 Nov 22 20:32 000000001c029c54.eyetvi -rw-r--r-- 1 stephenb staff 2537 Nov 22 20:32 000000001c029c54.eyetvr -rw-r--r-- 1 stephenb staff 3557421728 Nov 22 20:32 000000001c029c54.mpg -rw-r--r-- 1 stephenb staff 159802 Nov 22 20:32 000000001c029c54.thumbnail.tiff -rw-r--r-- 1 stephenb staff 2765146 Nov 22 20:32 000000001c029c54.tiff -rw-r--r--@ 1 stephenb staff 13091807 Nov 22 20:32 busy_000000001c029c54.iPad.m4v
A little later in the iPad Export:
total 7654600 -rw-r--r-- 1 stephenb staff 4330 Nov 22 20:32 000000001bf61603.eyetvp -rw-r--r-- 1 stephenb staff 316160 Nov 22 20:32 000000001c029c54.eyetvi -rw-r--r-- 1 stephenb staff 2537 Nov 22 20:32 000000001c029c54.eyetvr -rw-r--r-- 1 stephenb staff 120554 Nov 22 20:37 000000001c029c54.log -rw-r--r-- 1 stephenb staff 3390 Nov 22 20:34 000000001c029c54.logo.txt -rw-r--r-- 1 stephenb staff 3557421728 Nov 22 20:32 000000001c029c54.mpg -rw-r--r-- 1 stephenb staff 159802 Nov 22 20:32 000000001c029c54.thumbnail.tiff -rw-r--r-- 1 stephenb staff 2765146 Nov 22 20:32 000000001c029c54.tiff -rw-r--r-- 1 stephenb staff 0 Nov 22 20:34 000000001c029c54.txt -rw-r--r--@ 1 stephenb staff 358333351 Nov 22 20:37 busy_000000001c029c54.iPad.m4v
Even later in the iPad Export (almost done):
total 9111920 -rw-r--r-- 1 stephenb staff 4330 Nov 22 20:32 000000001bf61603.eyetvp -rw-r--r-- 1 stephenb staff 66 Nov 22 20:37 000000001c029c54.edl -rw-r--r-- 1 stephenb staff 316160 Nov 22 20:32 000000001c029c54.eyetvi -rw-r--r-- 1 stephenb staff 2781 Nov 22 20:37 000000001c029c54.eyetvr -rw-r--r-- 1 stephenb staff 396333 Nov 22 20:37 000000001c029c54.log -rw-r--r-- 1 stephenb staff 3390 Nov 22 20:34 000000001c029c54.logo.txt -rw-r--r-- 1 stephenb staff 3557421728 Nov 22 20:32 000000001c029c54.mpg -rw-r--r-- 1 stephenb staff 159802 Nov 22 20:32 000000001c029c54.thumbnail.tiff -rw-r--r-- 1 stephenb staff 2765146 Nov 22 20:32 000000001c029c54.tiff -rw-r--r-- 1 stephenb staff 112 Nov 22 20:37 000000001c029c54.txt -rw-r--r-- 1 stephenb staff 220 Nov 22 20:37 busy_000000001c029c54.iPad.chapters.txt -rw-r--r--@ 1 stephenb staff 1104194666 Nov 22 20:47 busy_000000001c029c54.iPad.m4v
AppleTV Export:
total 9389744 -rw-r--r-- 1 stephenb staff 4330 Nov 22 20:32 000000001bf61603.eyetvp -rw-r--r-- 1 stephenb staff 66 Nov 22 20:37 000000001c029c54.edl -rw-r--r-- 1 stephenb staff 316160 Nov 22 20:32 000000001c029c54.eyetvi -rw-r--r-- 1 stephenb staff 2781 Nov 22 20:37 000000001c029c54.eyetvr -rw-r--r--@ 1 stephenb staff 1246440219 Nov 22 20:49 000000001c029c54.iPad.m4v -rw-r--r-- 1 stephenb staff 396333 Nov 22 20:37 000000001c029c54.log -rw-r--r-- 1 stephenb staff 3390 Nov 22 20:34 000000001c029c54.logo.txt -rw-r--r-- 1 stephenb staff 3557421728 Nov 22 20:32 000000001c029c54.mpg -rw-r--r-- 1 stephenb staff 159802 Nov 22 20:32 000000001c029c54.thumbnail.tiff -rw-r--r-- 1 stephenb staff 2765146 Nov 22 20:32 000000001c029c54.tiff -rw-r--r-- 1 stephenb staff 112 Nov 22 20:37 000000001c029c54.txt -rw-r--r-- 1 stephenb staff 220 Nov 22 20:37 busy_000000001c029c54.iPad.chapters.txt
A little later in the AppleTV Export:
total 9389752 -rw-r--r-- 1 stephenb staff 4330 Nov 22 20:32 000000001bf61603.eyetvp -rw-r--r-- 1 stephenb staff 66 Nov 22 20:37 000000001c029c54.edl -rw-r--r--@ 1 stephenb staff 7 Nov 22 20:50 000000001c029c54.exported_inodes.txt -rw-r--r-- 1 stephenb staff 316160 Nov 22 20:32 000000001c029c54.eyetvi -rw-r--r-- 1 stephenb staff 2781 Nov 22 20:37 000000001c029c54.eyetvr -rw-r--r--@ 1 stephenb staff 1246440219 Nov 22 20:49 000000001c029c54.iPad.m4v -rw-r--r-- 1 stephenb staff 396333 Nov 22 20:37 000000001c029c54.log -rw-r--r-- 1 stephenb staff 3390 Nov 22 20:34 000000001c029c54.logo.txt -rw-r--r-- 1 stephenb staff 3557421728 Nov 22 20:32 000000001c029c54.mpg -rw-r--r-- 1 stephenb staff 159802 Nov 22 20:32 000000001c029c54.thumbnail.tiff -rw-r--r-- 1 stephenb staff 2765146 Nov 22 20:32 000000001c029c54.tiff -rw-r--r-- 1 stephenb staff 112 Nov 22 20:37 000000001c029c54.txt -rw-r--r-- 1 stephenb staff 220 Nov 22 20:37 busy_000000001c029c54.iPad.chapters.txt
Everything done:
total 9389752 -rw-r--r-- 1 stephenb staff 4330 Nov 22 20:32 000000001bf61603.eyetvp -rw-r--r-- 1 stephenb staff 66 Nov 22 20:37 000000001c029c54.edl -rw-r--r--@ 1 stephenb staff 7 Nov 22 20:50 000000001c029c54.exported_inodes.txt -rw-r--r-- 1 stephenb staff 316160 Nov 22 20:32 000000001c029c54.eyetvi -rw-r--r-- 1 stephenb staff 2781 Nov 22 20:37 000000001c029c54.eyetvr -rw-r--r--@ 1 stephenb staff 1246440219 Nov 22 20:49 000000001c029c54.iPad.m4v -rw-r--r-- 1 stephenb staff 396333 Nov 22 20:37 000000001c029c54.log -rw-r--r-- 1 stephenb staff 3390 Nov 22 20:34 000000001c029c54.logo.txt -rw-r--r-- 1 stephenb staff 3557421728 Nov 22 20:32 000000001c029c54.mpg -rw-r--r-- 1 stephenb staff 159802 Nov 22 20:32 000000001c029c54.thumbnail.tiff -rw-r--r-- 1 stephenb staff 2765146 Nov 22 20:32 000000001c029c54.tiff -rw-r--r-- 1 stephenb staff 112 Nov 22 20:37 000000001c029c54.txt -rw-r--r-- 1 stephenb staff 220 Nov 22 20:37 busy_000000001c029c54.iPad.chapters.txt
Thank you very much—that's very helpful!
But I'm a little disappointed not to see a file file that looks something like busy_000000001c029c54.AppleTV.m4v
inside the .eyetv
directory during the Apple TV export. I was hoping to use the presence of the busy_
files to indicate that transcoding isn't finished yet.
Just to sanity check: your Apple TV workflow finishes without an Apple TV recording in the .eyetv
directory, but such a file appears in iTunes somewhere?
I need to think about this a little more to get an approach that is robust across different workflows.
Would you please do me one more favor and tell me the name of the iTunes .m4v
file _during_ EyeTV export? If it's something like busy_Program-Name.m4v
, then this approach may yet work.
Yes, the file does wind up in iTunes. I was surprised not to see that file there too.
I may have to wait till tomorrow to do another test and see if I can catch the file as it's being sent to iTunes, but I'll see what I can do!
@Brandonshire, Thanks again. This would all be a lot easier if EyeTV offered some sort of API for monitoring progress, but afaik it does not. This is the reason for trying to infer EyeTV export status through UNIX commands. You can see that even Elgato faces this with their own triggered scripts, e.g. for their iTunes interface: "delay 5 --if the script does not seem to work, try increasing this delay slightly.
".
I believe it will be possible to refactor the existing scripts so that they are robust across workflows, but I'll need some help with different setups than my own.
When you do this check, would you also run these commands from the terminal? The recording will not appear in iTunes, so you must use the cd
command in Terminal to find the active export file. Right-click Show in Finder
from iTunes will get you most of the way there, and you can cut-and-paste Finder full path names into the Terminal.
_During_ an Apple TV Transcoding/Export, run:
lsof ~Shared/Public/iTunes\ Media/Movies/A\ Few\ Good\ Men.m4v
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME EyeTV 70515 username 34u REG 1,13 2261778432 24914147 /Volumes/Macintosh HD/Users/Shared/Public/iTunes Media/Movies/A Few Good Men.m4v
_After_ an Apple TV Transcoding/Export, run:
/Library/Application\ Support/ETVComskip/bin/mp4info ~Shared/Public/iTunes\ Media/Movies/A\ Few\ Good\ Men.m4v
/Library/Application Support/ETVComskip/bin/mp4info version 2.0.0 /Volumes/Macintosh HD/Users/Shared/Public/iTunes Media/Movies/A Few Good Men.m4v: Track Type Info 1 video H264 Main@3.1, 11037.437 secs, 4776 kbps, 1280x720 @ 29.970001 fps 2 audio MPEG-4 AAC LC, 11037.418 secs, 128 kbps, 48000 Hz 4 text Name: A Few Good Men Artist: Rob Reiner Release Date: 1992-01-01 Genre: TV Shows Comments: Best Picture nominee about three Navy lawyers charged with defending two marines accused of killing a third during a disciplinary action at the U.S. base in Guantanamo, Cuba. Cover Art pieces: 1 Media Type: Movie TV Network: AMC HDTV (East) Short Description: Best Picture nominee about three Navy lawyers charged with defending two marines accused of killing a third during a disciplinary action at the U.S. base in Guantanamo, Cuba. Long Description: Best Picture nominee about three Navy lawyers charged with defending two marines accused of killing a third during a disciplinary action at the U.S. base in Guantanamo, Cuba.
Please post your results as I have. It's important to get the full path and file name correct _during_ export, and show the results of the lsof
command on this open file.
My plan is to refactor the triggered scripts to use file activity and mp4info
for synchronization to fix the issues discussed above. This is a largish task and life is very busy so all I can say is I'll post a fix as soon as I am able to recode and throw some rocks at the fixes.
Thanks, I'll do what I can to get you the info you need! Thanks again for all of your hard work!
And yeah, totally agree about EyeTV needing to offer API's and such. Really I have to admit though it's the best option on the Mac (that I'm aware of at least), it's really starting to feel long in the tooth. I keep hoping someone else (or Elgato themselves!) will come out with a product for the Mac that can easily replace it, and start with some more modern ideas about how everything should work in DVR software.
Please try the latest release, v3.4, and let us know how it goes.
I made several improvements to the Triggered Scripts, and honestly had to go back and rtfm EyeTV's applescript dictionary, which provides all the desired API functionality, specifically the is_exporting
test. This also allows nice pythonesque features like appscript.reference.app('EyeTV').is_exporting()
.
If there's a Mac-based PVR setup that comes close to EyeTV, I'm not aware of it. Which is one reason I don't mind coding for EyeTV.
Happy Thanksgiving. Must go switch from code to brine.
Ok, back from Thanksgiving and now I've had a little time to check things out and do some testing. So good news! It seems everything is working as expected for scheduled shows that are automatically exported to iTunes! For things that I've scheduled and told EyeTV to automatically export to iTunes, the file that's going into iTunes has all of the content and commercials and it has chapter markers around the commercials so they can be skipped by advancing a chapter, and if it's not quite accurately marked I can easily fast forward or rewind a little bit and not miss any content. The is is working with the "Export to AppleTV 1080p" option set as discussed above. So it's making an iPad version and then automatically exporting an iTunes version, and things are working as you would hope! So things are working the way they were before with the old (WINE based) set up! Yay!
One tip I might add for others is that initially after installing the new build (and logging out and back in) things weren't working at all. Commercials weren't being marked at all (looking at the recording in EyeTV had not commercials marked, and of course exports didn't have chapter markers for where the commercials were. However, after I uninstalled the program, (I actually ran the uninstaller for both the new version and the older one I had installed on top of, not sure if that was necessary but seemed worth doing), I then reinstalled them and restarted the computer (I'd have logged out and back in but had something else that required a restart anyway). When I got back in I had a few test recordings go and everything worked as expected!
However! I am still in the position where if I manually export something from EyeTV to iTunes, at least using the AppleTV 1080p option, I am still getting commercials cut out, not just marked as chapters. Is there any way to fix this? This was also the way it worked in the old WINE set up, but I had always meant to look into whether there was a way to make manual exports work the same way that automatic ones do. (occasionally I might want to go back and watch something on the AppleTV again that I've already deleted from iTunes, or something might be a one-off recording that I don't want to set up a smart guide for, which is the only way I've figured to have things automatically exported to iTunes).
If i export the iPad version (meaning I go into preferences and change what the export button does) to iTunes I will get a version with chapter markers around commercials rather than them having been cut, so I guess that's a work-around but it's a little bit of a pain to have to change that around every time I want to manually export something, and then change it back after it's done (of course the plus side there is that the iPad export is very fast since it's an already existing file).
Great! Glad that your workflow works now.
The deletion of marked commercials for post-comskip hand exports is expected behavior because EyeTV deletes the content specified in .eyetvr
via the comskip .edl
edit decision list.
I've thought about adding the feature of a MarkCommercials --unmark
flag that would delete the markings, but it's honestly faster, simpler, and easier to do this by hand with a right-click on the recording in the rare cases I need hand exports.
Thanks for all your work on this! It's very much appreciated!
So I was hoping just for the opposite. I would love the commercials to be cut when I automatically export to itunes. I have the same workflow where eyetv records using a smart guide, then exports the recording automatically to itunes with "AppleTV HD 1080P."
Is there an option to make it cut the commercials when it exports? I am using the new Appletv to watch the videos and it does not seem to see the chapter markers so I can easily skip commercials.
It's a lot more reliable to leave in the ads and use helper software to skip through them. This is because as awesome as Comskip is, its false alarm/missed detection/alignment isn't perfect, so it avoids annoyances to leave the final adjustments to the user.
The helper software is the ComSkipper script for EyeTV on OS X . On iOS devices the native Videos app honors mp4 chapter skipping imbedded with the mp4chaps command. This extends to external device control like Apple earphones.
If you still want to strip commercials out of the stream, just use EyeTV to transcode and export recordings after they've been marked by comskip.
If you want to automate this, recode the AppleScripts RecordingDone and ExportDone so that exports wait for comskip to finish, rather than comskip waiting for exports to finish.
Also, Elgato's Turbo.264 software honors .edl files and does this automatically, with the ability to hand-adjust the .edl yourself.
I am using the new Appletv to watch the videos and it does not seem to see the chapter markers so I can easily skip commercials.
That's annoying. Please send Apple a feature request that the forward and backward arrows on tvOS support chapter skipping, just like on iOS.
As instructed over at the Apple Support forums I am opening an issue here.
I am having an issue in which recordings that are automatically exported from EyeTV to iTunes seem to, at least sometimes, have the commercials cut rather than simply marked.
Right now my recordings are all governed by smart-guides which are set to automatically export to iTunes. Most of the time in the past it would simply export it with commercials marked, but not cut, last night we found a recording from the other day HAD been automatically exported to iTunes, (there was no intervention from me) but the commercials had been cut. Again, I did not export by hand, this was the automatic iTunes export.
So at this point I'm trying to figure out why this happened, and how to make sure it doesn't happen again. I really like having commercials marked, even if occasionally it's not perfect, but I DON'T want them just cut out of the recordings, as there are often false positives and times where a few minutes of actual programming will get cut.