Crash on launch when accessing thetvdb.com

[talientinc]

I see that @rrwclc posted a crash in Issue #476 about an hour ago.

Originally posted by @rrwclc in https://github.com/mackworth/cTiVo/issues/476#issuecomment-1120015978

I have been getting crashes on startup since the middle of this afternoon (May 6th, 2022). I suspect that the issue is related to handling the responses from the queries to thetvdb.com.

I am running cTiVo on both a new MacStudio using the v3.5.0 Beta software and also on an 2018 Mac Mini using v3.4.3 (I think) software. I had been using cTiVo rather heavily over the past couple of days re-downloading a fair number recordings, so I know it had been working cleanly. I bounced the application around 11:30 AM MDT (in an attempt to get it to refresh the list of TiVo's) and it failed on startup. Trying to debug what was going on, I worked through several scenarios.

Thinking it might have been a corrupt cache, I purged the cache at ~/Library/Caches/com.cTiVo.cTiVo. No good. Perhaps it was a corrupt preferences file, so I restored a copy from last night. No good. I moved over to the old Mac Mini and started the app there. I hadn't started cTiVo up on that machine since I migrated to the new Mac Studio as my primary work machine, so it would not have had a freshly corrupted cache or preference file. It crashed on startup, too.

That pointed to something external to the cTiVo installation itself. I bounced the TiVo's on the network in case they were sending some sort of poison pill in response to populating or refreshing the Now Playing window. Still crashing.

Looking at the logs, the most recent entries were all related to gathering information from thetvdb.com. I unplugged the connection between the router and the cable modem. The application started cleanly. When I re-established internet access, it crashed immediately on startup. I confirmed this behavior with a couple of cycles of testing. I also went to the advanced preferences and cleared the cache (and verified that the information did update in the preferences file). It didn't help. If anything, it made it crash more quickly on startup.

Looking at the logs, it looks like at least some of the tvdb queries are being received and processed cleanly, but it feels like there must be some new category of responses from their server that are triggering the core dump.

[mackworth]

Yup. I'm seeing the same problem. Trying to track it down.

[mackworth]

Looks like they fixed it. Are you still crashing?

[mackworth]

Looks good; my 3.5.0 copy hasn't crashed on multiple launches (versus every time before), and haven't seen a field crash for the last half hour and theTVdb files are updating properly.

FYI, on request for an episode's details, they were sending { data = "<null>"; links = "<null>"; } as opposed to { error = "whatever" } that they're supposed to send. Given there was no error, cTiVo was then looking for a subfield inside data w/o checking that it was non-null. I'll go ahead and include a fix in a 3.5.1 to log the problem, instead of crashing.

BTW, thank you for the Excellent report! Having such clear info really helped me confirm the problem.

[talientinc]

Sorry, it still crashed for me just now

Date/Time: 2022-05-06 17:44:06.8950 -0600:

Crashed Thread: 1 Dispatch queue: com.apple.NSURLSession-delegate

Perhaps they've rolled out the fix to a certain number of instances of their cluster, but it hasn't deployed to all of them yet.

I did see that same null value in my logs and thought it was a likely candidate to be causing the issue.

Anyway, I'll check again after dinner to see if it clears up for me.

[talientinc]

Also, I forgot that I had left the TVDB logs in verbose, so I can still see the problematic entry still showing up as of five minutes ago:

2022-05-06 17:50:41:078 -[MTTVDB searchForEpisodeNamed:orSeason:andEpisode:inSeries:completionHandler:failureHandler:pageNumber:]_block_invoke@314>TVDB Episode JSON: {
    data = "<null>";
    links = "<null>";
}

[mackworth]

So, I see four crashes on a Mac mini, so I assume that's @talientinc. I've posted 3.5.1; can you try it ASAP? (As I no longer have a bad server to test against?)

[mackworth]

https://github.com/mackworth/cTiVo/releases/download/3.5.1/cTiVo.app.zip

[talientinc]

No more crashes with v3.5.1 and I do see multiple log entries of this nature:

2022-05-06 18:53:57:751 -[MTTVDB searchForEpisodeNamed:orSeason:andEpisode:inSeries:completionHandler:failureHandler:pageNumber:]_block_invoke_2@356>TVDB Episode JSON type Error (no links data): 71969 for https://api.thetvdb.com/series/71969/episodes/query?page=1

Thank you very much! I appreciate all the work you've done over the years on this.

[mackworth]

Great to know the patch works. I got one run reporting the errors before server started working again. Let me know if you don’t start getting real data.

[rrwclc]

Processing transfers tonight, will see if I have any issues.

Doing a default, 1080, and High Quality conversions.

Thank you for the quick fix!

-Ronald

From: Hugh Mackworth @.> Sent: Friday, May 6, 2022 8:24 PM To: mackworth/cTiVo @.> Cc: rrwclc @.>; Mention @.> Subject: Re: [mackworth/cTiVo] Crashing when accessing thetvdb.com? (Issue #477)

Great to know the patch works. I got one run reporting the errors before server started working again. Let me know if you don’t start getting real data.

— Reply to this email directly, view it on GitHub https://github.com/mackworth/cTiVo/issues/477#issuecomment-1120102174 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AZBXCLGVPHTBVPYXSFWXD4LVIXA4RANCNFSM5VJN236Q . You are receiving this because you were mentioned.Message ID: @.***>

[talientinc]

FWIW, I am still seeing the no links responses from thetvdb.com as of a few minutes ago (time stamp is in Mountain time). This was after a full reboot (for reasons unconnected to cTiVo). This doesn't have any adverse impact on what I am doing, so I am in good shape. Thanks!

2022-05-07 13:52:20:434 -[MTTVDB searchForEpisodeNamed:orSeason:andEpisode:inSeries:completionHandler:failureHandler:pageNumber:]_block_invoke_2@356>TVDB Episode JSON type Error (no links data): 417698 for https://api.thetvdb.com/series/417698/episodes/query?page=1

[rrwclc]

I’ve done a number of transfers since the update with no issues.

I’m so appreciative of the fix!

Sent from my iPhone

On May 7, 2022, at 15:02, talientinc @.***> wrote:

 FWIW, I am still seeing the no links responses from thetvdb.com as of a few minutes ago (time stamp is in Mountain time). This was after a full reboot (for reasons unconnected to cTiVo). This doesn't have any adverse impact on what I am doing, so I am in good shape. Thanks!

2022-05-07 13:52:20:434 -[MTTVDB @.***>TVDB Episode JSON type Error (no links data): 417698 for https://api.thetvdb.com/series/417698/episodes/query?page=1 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[mackworth]

I've pushed 3.5.1 via sparkle and filed a support ticket with theTVDB asking why some are still getting the bad data.

Thanks to both for reporting this; for some reason, I didn't get a notification from Crashlytics that there was a sudden surge, so I wouldn't have known about it otherwise.

I'll leave this open for a few days for others checking for recent issues.

[mackworth]

@talientinc are you still seeing errors in the log? I’m not seeing any crashes, although I won’t if they upgrade.

[talientinc]

Yes, still.

2022-05-08 19:56:59:224 -[MTTVDB searchForEpisodeNamed:orSeason:andEpisode:inSeries:completionHandler:failureHandler:pageNumber:]_block_invoke_2@356>TVDB Episode JSON type Error (no links data): 338736 for https://api.thetvdb.com/series/338736/episodes/query?page=1

Hmm, the (verbose) log message immediately preceding the above message does show the links/next path to be populated in the response.

Here is that message with the data array elided for brevity:

2022-05-08 19:56:59:224 -[MTTVDB searchForEpisodeNamed:orSeason:andEpisode:inSeries:completionHandler:failureHandler:pageNumber:]_block_invoke@314>TVDB Episode JSON: {
    data =     (

    );
    links =     {
        first = 1;
        last = 2;
        next = 2;
        prev = "<null>";
    };
}

If I drop that URL into Safari, I see the same response:

{"links":{"first":1,"last":2,"next":2,"prev":null},"data":[...]}

In the updated code, I see you added what appears to be a strict equality comparison ([links class] == [NSDictionary class]) where elsewhere in the call it appears that you are generally checking types using isKindOfClass. I have never programmed in Objective-C, so I don't know the details of the techniques for type checking. Perhaps the JSON deserializer wrapped the links element in a subtype of NSDictionary or something along those lines so that the equality comparison operator is failing?

[rrwclc]

Not related to this issue and maybe another ticket / issue posted…

I noticed that some videos I transfer the audio isn’t in sync. What am I doing wrong? I have a MacBook Pro 2020 loaded ram, i7.

Sent from my iPhone

On May 8, 2022, at 21:43, talientinc @.***> wrote:

 Yes, still.

2022-05-08 19:56:59:224 -[MTTVDB @.***>TVDB Episode JSON type Error (no links data): 338736 for https://api.thetvdb.com/series/338736/episodes/query?page=1 Hmm, the (verbose) log message immediately preceding the above message does show the links/next path to be populated in the response.

Here is that message with the data array elided for brevity:

2022-05-08 19:56:59:224 -[MTTVDB @.***>TVDB Episode JSON: { data = (

);
links =     {
    first = 1;
    last = 2;
    next = 2;
    prev = "<null>";
};

} If I drop that URL into Safari, I see the same response:

{"links":{"first":1,"last":2,"next":2,"prev":null},"data":[...]} In the updated code, I see you added what appears to be a strict equality comparison ([links class] == [NSDictionary class]) where elsewhere in the call it appears that you are generally checking types using isKindOfClass. I have never programmed in Objective-C, so I don't know the details of the techniques for type checking. Perhaps the JSON deserializer wrapped the links element in a subtype of NSDictionary or something along those lines so that the equality comparison operator is failing?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[mackworth]

@talientinc You're exactly right, and it's stupider than even that makes it sound.

NSDictionary is a cluster of classes that the ObjC runtime picks from to optimize performance. You can't do the equal test ever. And ... I always forget. Particularly when doing a hotpatch...

5.0.2 posting soon.

[mackworth]

@rrwclc Unfortunately not much I can do about that. It's probably due to some minor glitch in the video stream, and TiVo's code for encrypting the stream has decayed over the years. They don't seem to care.

[mackworth]

@talientinc 3.5.2 is posted with that fix. Thanks again for your help (and impressive ObjC diagnosis).

[talientinc]

I was surprised to see that I was still seeing the same behavior with the new release. Long story short, it appears that the ZIP file in the v3.5.2 release contains a copy of the v3.5.1 binary, whose time stamp is May 6, 2022 at 6:00 PM (as reported on my Mountain Time machine).

[mackworth]

Well, not entirely clear how I managed that, but I've actually uploaded 3.5.2 this time.

[talientinc]

It all looks clean now.

I emptied the cache to trigger reloading everything. No more log messages about missing links elements. I also verified that the code is successfully navigating to the second and subsequent pages of links as necessary to search for information. (The furthest I saw it go was 10 pages down the list, though there might be some deeper searches in the log files that rolled off because verbose logging over a complete refresh of four mostly full TiVo units with recordings dating back years generates a tremendous amount of volume.)

Thank you very much!

[mackworth]

Great; sorry for the mistakes. Thanks for your help!

And thanks for checking that linkage test in particular; I don't think I've done that check in years.

Now the only problem is that TVDB plans to switch one of these days to a paid subscription, so everyone who wants to use it will have to pay them $12/year. Given that it's a backup for TiVo data, which has gotten better recently AFAICT, I'm guessing not many will do so, yet I'll have to rewrite the API calls, add a UI for the subscription...