3.3.0 Beta Issues

[mackworth]

A place to discuss the 3.3.0 beta release

[mackworth]

Please update to Beta2, as I found a case in Beta1 where it wasn't properly labeling H264 files...)

[tannebil]

Any suggestions as to channels that might be mixing H.264 and MPEG2 files so that I can test against them? Is there an easy way to see what's going on in the channel other than digging through the log files?

[mackworth]

Great question. In my recent testing, I see the mixed phenomenon primarily on the HD versions of local channels (e.g. PBS: KOPBDT and CW: KRCWDT here on Comcast Portland).

Easiest way to check it is to use the Test PS format. Now the "Test Selected Channels" button in Edit Channels will only do one show per channel, but you can do it manually:

1) In Edit>Edit Formats, select Test PS, click "Show in User Interface" and then "Save". 2) In Preferences>General, turn off: Add Captions, Cut Commercials and Mark Commercials. 3) Select the shows you'd like to test, e.g. all on one channel, or just all of them. I'd turn off Folders, turn on Suggestions, and sort by Channel. 4) Hit Download (or Add to Queue), and let it run. It's very quick per show, but it has to wait a minute in between to not upset the TiVo downloader, so it'll do 50-60 shows per hour. 5) Each tested show will now have either -- for MPEG2 or a ✔(dark check) in the H.264 column. Sort by Station, and you'll see any mixed channels.

@astettner Exactly: Most. The previous versions assumed that each channel would migrate from all MPEG2 to all H264. What I'm seeing is that's true for most channels, but not all. The previous algorithm would see H.264 (which always fails over PS), and switch all future downloads to TS, which sometimes corrupts MPEG2 downloads. Now it also checks for that other direction MPEG2 over TS and downloads the show properly over PS. So if a channel is one or the other, no problem; if it has one show the other way, then it will switch, but then switch back. If it stays mixed (as I"m seeing sometimes), it will use the last show it downloaded as a reference for how to try the next one, but will retry if that was wrong.

[mackworth]

Um, yes; it’s here: https://github.com/mackworth/cTiVo/issues/354#issuecomment-465432798

It’s indeed public, but you can edit any comments that you don’t want to remain there...

[tannebil]

I ran it against my existing recordings for my local PBS and Fox channels. In the case of the PBS station (KCTS), the channel preferences list doesn’t show it as an H.264 station and the “Use TS” has a dash in it. The H.264 column on the TiVo list shows a mix of “—“ and bold checkmarks. In the download list, it shows the same. All of these shows had been previously downloaded and encoded without an issue that I can recall. I checked one that was H.264 and one that was not but didn’t see any issue with either. Five of the twenty-three downloads failed. All ran successfully when rescheduled. Five of the shows show as H.264 (no overlap with the ones that failed)

The Fox channel (KSTW) show both the “Use TS” and “H.264” boxes clear. Four of fourteen downloads failed (two were successful on the first reschedule, the other two failed every reschedule). Two of the successful downloads show H.264 (once again a different two than the failures). Both the the failed shows are in my iTunes library so they must have downloaded OK when I first recorded them. I’m redoing the downloads with the Default format overnight.

I’ll leave it to you to interpret the results vs the expected results as I quickly get lost in the complexities of the process.

[mackworth]

So, for this purpose, ignore the channel preferences info. (it simply reflects the last seen show) (And the H264 column is the exact same info for downloads/show table

So you're seeing the mixed mode both MP2 and H264 on both channels, confirming my local testing. My impression based on what I've seen is that many MP2s over TS work fine, some will be a little corrupted (missing frames, audio glitches), and others will be damaged enough that ffmpeg gives up entirely. Because so many make it through is why we haven't seen a big problem: this issue only applies to mixed channels, and MP2s over TS usually are ok, or at least report ok.

When you say "five of twenty three" and "four of fourteen failed", I take it that's Test PS downloads. That's surprising. (Last night, I ran 1300 Test PSs without a failure.) I'd love to see that logs on those, especially as they were ok for regular downloads. Maybe mencoder is failing for you, and I should switch to ffmpeg.

[tannebil]

I'll rerun them tonight and send you the logs if it happens again. Any particular log settings you want me to use?

On Feb 20, 2019, at 11:12 AM, Hugh Mackworth notifications@github.com wrote:

So, for this purpose, ignore the channel preferences info. (it simply reflects the last seen show) (And the H264 column is the exact same info for downloads/show table

So you're seeing the mixed mode both MP2 and H264 on both channels, confirming my local testing. My impression based on what I've seen is that many MP2s over TS work fine, some will be a little corrupted (missing frames, audio glitches), and others will be damaged enough that ffmpeg gives up entirely. Because so many make it through is why we haven't seen a big problem: this issue only applies to mixed channels, and MP2s over TS usually are ok, or at least report ok.

When you say "five of twenty three" and "four of fourteen failed", I take it that's Test PS downloads. That's surprising. (Last night, I ran 1300 Test PSs without a failure.) I'd love to see that logs on those, especially as they were ok for regular downloads. Maybe mencoder is failing for you, and I should switch to ffmpeg.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[bg3dev]

Here are the new logs from 3.3 beta 2 for the issue where cTivo doesn't see the classic TivoHD on the network. I believe one other user reported this with the last beta. Hope this helps.

Kinda wondering if it's the apostrophe in the name of the Tivo -- If you can tell me how to rename the Tivo, I can try this.

com.cTiVo.cTiVo 2019-02-23 02-17.log.zip

[mackworth]

I think I've found the problem. Can you run this program called Discovery (the former Bonjour Browser)?

if you could expand all of the sections starting with _tivo (just one level expansion needed) and then post a screenshot of those sections, that would be great.

[mackworth]

@bg3dev and @wjmt3 (and anyone else with an TiVo HD or older) can you try this version a try?

https://www.dropbox.com/s/5zk06yivmsixkka/cTiVo.app.zip?dl=0

No need to do anything during startup, this version has some extra logging. Just post the log after it runs. Thanks.

First question is whether it recognizes it and loads up the Now playing list; second question is whether everything else runs fine with it after that?

[wjmt3]

My oldest TiVo is currently a Premiere. Didn't that come after the HD?

On 02/24/2019 12:01 pm, Hugh Mackworth wrote:

@bg3dev [1] and @wjmt3 [2] (and anyone else with an TiVo HD or older) can you try this version a try?

https://www.dropbox.com/s/5zk06yivmsixkka/cTiVo.app.zip?dl=0 [3]

No need to do anything during startup, this version has some extra logging. Just post the log after it runs. Thanks.

First question is whether it recognizes it and loads up the Now playing list; second question is whether everything else runs fine with it after that?

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [4], or mute the thread [5].

Links:

[1] https://github.com/bg3dev [2] https://github.com/wjmt3 [3] https://www.dropbox.com/s/5zk06yivmsixkka/cTiVo.app.zip?dl=0 [4] https://github.com/mackworth/cTiVo/issues/354#issuecomment-466811300 [5] https://github.com/notifications/unsubscribe-auth/AZx7ERN7U0UCIyiC5dD6Kfpz8FbDhbINks5vQu-dgaJpZM4a_NAM

[mackworth]

The Premiere (Model 4) is indeed newer than the HD (Model 3). Still, if you’re still not seeing your Premiere, give this one a try. And if it doesn’t work, if you could run the Discovery program above, that would be good too. Thx

[bg3dev]

The dropbox version works. I got a login keychain prompt when opening it, and after allowing access to the keychain everything populated. Still need logs?

[mackworth]

Update to Beta 3; first release candidate.

[bg3dev]

Out of curiosity — what changed between Beta 2 and Beta 3 that made the new build work on older Tivo’s again? I looked at the code diff and couldn’t spot it. Thanks again!

[mackworth]

It's a tiny change. It's in MTIVoManager. line 512: -(void)searchForBonjourTiVos: [tivoBrowser searchForServicesOfType:@"_http._tcp" inDomain:@"local."]; vs [tivoBrowser searchForServicesOfType:@"_tivo.device._tcp" inDomain:@"local."];

TiVos broadcast multiple services over Bonjour (aka mDNS). Most are _tivo-xxx where xxx is one of device, mindrpc, remote, videos, videostream and xcode. Not quite clear what all of those means (e.g. xcode?). We used to use _tivo-videos, but when I added Remote controls, Minis don't advertise that (as you can't download from a Mini), so I changed to _tivo-device. Unfortunately, the older TiVos (which I no longer have) don't advertise that, so they no longer got picked up. I changed to the generic web server (_http) as that's the only one I can see that all three kinds use (and it's annoying to run two separate search for two different names). That also picks up every other web-interface devices, but those get filtered out in netServiceDidResolveAddress, as they don't have Tivo Serial Numbers (TSN key).

[ghost]

Hello...minor issue (perhaps cosmetic)...I am trying to download a simple decrypted Tivo file, and have set AutoSkipMode retrieval off in the preferences. The SkipMode box remains checked in the download queue line, and it appears to make a pass prior to downloaded to find the SkipMode entries.

[mackworth]

Good question; I've tried hard to simplify the options, but it's still confusing.

Auto SkipMode only indicates whether cTiVo will do the SkipMode process by itself (during indicated times) or whether it will wait for you to do it manually. It doesn't change whether a Download gets marked as needing SkipMode. That is set by the Commercials Strategy pulldown. If that is set to anything other than comskip only, then it will mark a new Download as needing SkipMode.

So, in that case (auto-SkipMode off, and a Download needing SkipMode), the expected behavior is that the Download should just wait in the queue forever. It can be launched, either by "Get SkipMode from TiVo" (from either main menu or Cmd-click contextual menu), OR by just turning off SkipMode checkbox for that download.

On the other hand, if you did see cTiVo launching an unauthorized auto-SkipMode prior to download, then that's definitely a bug, and I'll want to see the logs (preferably with Download debug level set to Detail).

[mackworth]

Ah, just tried it with Decrypted TiVo Show. Even though SkipMode is marked, it will go ahead and immediately download, as it knows it can neither Cut nor Mark commercials with that Format, but I did not see it try and do the SkipMode

[ghost]

Ok...thanks for this.

Great piece of code...

jp

Sent from my iPhone

On Mar 3, 2019, at 12:21 PM, Hugh Mackworth notifications@github.com wrote:

Ah, just tried it with Decrypted TiVo Show. Even though SkipMode is marked, it will go ahead and immediately download, as it knows it can neither Cut nor Mark commercials with that Format, but I did not see it try and do the SkipMode

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[mackworth]

Thank you!

[mackworth]

And... the next version will not even set SkipMode in that case (visual effect only).

[tannebil]

I think I was just messed up on the work flow. Comskip used to only work with MPEG2 files so I assumed that you are just hacking around it somehow. However, since commercial marking is available on h.264 channels, Comskip must have been updated. Duh. but that means that (assuming I don’t care about the size), I should use Decrypt MP4 as my default for H.264 channels. Which raises the question as to what happens when an MPEG2 encoded show gets downloaded from that channel? Better give that a test.

On Mar 3, 2019, at 4:13 PM, Hugh Mackworth notifications@github.com wrote:

And... the next version will not even set SkipMode in that case (visual effect only).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mackworth/cTiVo/issues/354#issuecomment-469081129, or mute the thread https://github.com/notifications/unsubscribe-auth/AI31JKrGXmOCiwZA4aNt57OyJeexM3r0ks5vTGVHgaJpZM4a_NAM.

[ghost]

Wondering…how hard would it be to have an apple-compatible .h265 format?

I’ve tried to set up a custom one, using what I think the arguments are from the Plex post-processing script I use, and have failed miserably…of course, it may be that I didn’t point at an encoder that would do that (the ffmpeg I squirreled over in my /usr/local/bin…have to go think about that one some more…

jp

On Mar 3, 2019, at 2:03 PM, john post postjc@me.com wrote:

Ok...thanks for this.

Great piece of code...

jp

Sent from my iPhone

On Mar 3, 2019, at 12:21 PM, Hugh Mackworth <notifications@github.com mailto:notifications@github.com> wrote:

Ah, just tried it with Decrypted TiVo Show. Even though SkipMode is marked, it will go ahead and immediately download, as it knows it can neither Cut nor Mark commercials with that Format, but I did not see it try and do the SkipMode

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mackworth/cTiVo/issues/354#issuecomment-469060099, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2aC5n5h4a_B6EWg94NK1VtPjnoiN3sks5vTC66gaJpZM4a_NAM.

[mackworth]

@dapostman. See issue #343 for some discussion here

[mackworth]

Beta 4 Second release candidate

[wjmt3]

Downloaded. Just out of curiosity, what does DownloadDone.scpt do?

On 03/04/2019 9:56 pm, Hugh Mackworth wrote:

Beta 4 [1] Second release candidate

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [2], or mute the thread [3].

Links:

[1] https://github.com/mackworth/cTiVo/releases/tag/3.3.0beta4 [2] https://github.com/mackworth/cTiVo/issues/354#issuecomment-469549337 [3] https://github.com/notifications/unsubscribe-auth/AZx7Eb5EnbSUQFcMJ9S_3oGMpF4vQGvCks5vTgb6gaJpZM4a_NAM

[mackworth]

It announces your completed transfers via Pushover service. User request, but it’s been particularly useful for me when doing a long test; my Apple Watch chimes nicely when it is completed). A few others are using it as well.

(From release notes:) Notes on Pushover: Pushover is a real-time notification system. Due to user requests, cTiVo will now let you know when your shows have completed. To use it, you'll have to sign up with their system, which will give you a userId. Download the attached Applescript and place it in ~/Library/Application Scripts/com.cTiVo.cTiVo/. Don't change the name from DownloadDone.scpt. After the first successful download, the script will ask for your Pushover userId. (While this process is admittedly a little tedious, it only has to be done once, and Pushover reports back how many people are using the service. if a lot do, then I'll include turning on the script in the UI.)

More generally, you could use the same hook to do any post-processing on downloaded shows you wish. Documentation for now consists of looking at the Pushover script.

[mackworth]

Beta 5 is available. Unless something dramatic shows up, this will be final as reliability is much higher than current 3.1.2

Slowed down SkipMode retrieval a bit to improve reliability on slower TiVos Better SkipMode reporting in logs Allows drag/drop of show folders into Downloads or Subscriptions Avoids a case where simultaneous encoders goes above limit Fixes a rare random crash while updating progress

[wjmt3]

I just experienced an issue in downloading a show. After an initial period "waiting for TiVo", (thanks for the new progress bar!) a show began to download. However, when it got a short way through the download (by eyeball, maybe 15 percent) the status suddenly changed from "downloading" to "subtitling", and the download rate (which had been on the slow side) went away. It then gave a "detecting captions failed" error, and proceeded to go back to "downloading" – this time at a much faster rate. The download succeeded, and encoding is now underway.

But this is an issue I've never experienced on a non-beta release. Maybe it's an issue with the one specific recording, but just in case...

[mackworth]

I’d want to see the logs, but I think you’re ok. Did the progress bar restart entirely? If so, that suggests that it decided it was downloading in the wrong format, and so restarted. Which is proper behavior (other than the spurious caption failed msg, which I’ll look into)

[wjmt3]

It did restart entirely. I'll let you know whether captioning did indeed fail when encoding finishes.

[wjmt3]

The recording seems to be stuck. The progress bar is all the way across, saying "Wait SkipMode (Mark)." Encoding is done (fans are back to normal).

I'd cancel it, but that would probably lose the whole recording. (I don't care about SkipMode.)

[mackworth]

If you turn off Skipmode, it’ll finish up.

[wjmt3]

If you turn off Skipmode, it’ll finish up.

Ah, didn't see your reply before my last post. Thanks! Fortunately, I hadn't done anything yet.

[wjmt3]

Apparently, I can't seem to figure out how to accomplish that. I've gone into Preferences, unchecked Mark Commercials, set Strategy to ComSkip Only, and unchecked Enable Auto SkipMode Retrieval; nothing has changed in the status of the recording. Is there something else I have to do to get it to finish up?

[mackworth]

Sorry to be imprecise. In the download table,there should be a SkipMode column (vs Skipdata, which shows whether data is available). Cmd-click on the table header to add the column if it’s not visible now. If you turn that off, it will then finish up.

OR you could just right-click on the download and hit Load SkipMode info right away.

[wjmt3]

That worked, and the captioning failure message was indeed spurious. Thanks!

[mackworth]

After a year of development with 181 commits, 3.3.0 is finally released...

(No serious changes since beta5; a couple of minor fixes/better messages for iTunes compatibility under Mojave.)

So, closing this issue; any problems feel free to open a new one.

[mackworth]

So Crashlytics is reporting an increase in the Failure rate, particularly for Default with no commercials or captions. Unfortunately, due to the anonymization, I don't know anything beyond that. (e.g. I don't know if it's a couple of people having a problem, maybe due a particular cable system, or across the board) . I'm not seeing any problem on my system.

Is anyone in this early testing group seeing a significant change in the % of shows that fail? In particular, if you have a failure that then works with comskip or captions that would be startling and worth looking into.

[ghost]

Not here; but usage is low at the moment.

Sent from my iPhone

On Mar 26, 2019, at 6:31 PM, Hugh Mackworth notifications@github.com wrote:

So Crashlytics is reporting an increase in the Failure rate, particularly for Default with no commercials or captions. Unfortunately, due to the anonymization, I don't know anything beyond that. (e.g. I don't know if it's a couple of people having a problem, maybe due a particular cable system, or across the board) . I'm not seeing any problem on my system.

Is anyone in this early testing group seeing a significant change in the % of shows that fail? In particular, if you have a failure that then works with comskip or captions that would be startling and worth looking into.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[tannebil]

Hitting it hard and not seeing any failures at all. Running 80% Default with commercial marking.

On Mar 27, 2019, at 1:49 AM, dapostman notifications@github.com wrote:

Not here; but usage is low at the moment.

Sent from my iPhone

On Mar 26, 2019, at 6:31 PM, Hugh Mackworth notifications@github.com wrote:

So Crashlytics is reporting an increase in the Failure rate, particularly for Default with no commercials or captions. Unfortunately, due to the anonymization, I don't know anything beyond that. (e.g. I don't know if it's a couple of people having a problem, maybe due a particular cable system, or across the board) . I'm not seeing any problem on my system.

Is anyone in this early testing group seeing a significant change in the % of shows that fail? In particular, if you have a failure that then works with comskip or captions that would be startling and worth looking into.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[wjmt3]

Correct me if I am wrong, but I thought the problem only manifested without commercial marking (or captions)

[mackworth]

Not sure, as what I see is an increase in failures with Default, and also an increase in Failures on type 0 (no comskip or captions). Not necessarily at the same time, although that’s the easiest explanation.

[mackworth]

for those following along here, I've just posted beta1 for 3.3.1.

[ghost]

Hugh,

Appreciate the support from you for this code; very helpful.

jp

Sent from my iPhone

On Mar 28, 2019, at 3:55 PM, Hugh Mackworth notifications@github.com wrote:

for those following along here, I've just posted beta1 for 3.3.1.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.