trakt import fails at 999 imports

[samcro1967]

trakt import fails at 999 imports. I am using sqlite and running App version: 0.35.3.

Here is the log: movary-2022-12-23T15-41-23.log

[leepeuker]

The problem was the Trakt API rate limit (max 1000 request per 5min it seems). There are no safeguards implemented in movary for this yet. I will improve this. For the moment wait 5min and try again, the rate limit should have been reset then.

Note: This should only happen on initial or force imports with a big library. Still sucks and will be fixed, but normal day to day usage should not be affected.

[samcro1967]

I did rerun it this afternoon. It increased from 999 movies to 1630 and then died again. New log attached of the last attempt.

movary-2022-12-24T00-00-20.log

[leepeuker]

Thank you again, the second one contains a different bug (required movie cast data is missing) will fix this soon, too.

Edit: And merry christams :santa:

[leepeuker]

I did rerun it this afternoon. It increased from 999 movies to 1630 and then died again. New log attached of the last attempt.

movary-2022-12-24T00-00-20.log

This should have been fixed with 0.35.4, please try again :)

[samcro1967]

Increased from 1630 to 1632 and then stopped. Newest log attached.

movary-2022-12-25T13-02-31.log

[leepeuker]

Did you execute the database migration after updating? It think that is missing 🤔

[samcro1967]

I did not. :disappointed: Increased from 1635 to 1792. Should get to about 2300. New log attached. movary-2022-12-25T14-20-53.log

[leepeuker]

Ah, this time it fails because it is expected that movies all have a poster in tmdb, but it seems you have a movie without one. Going to fix this too.

I am sorry for all the problems you are encountering. Your big library is perfect to catch all these edge cases I did not think of or knew existed, so thank you again for taking the time to report these bugs!

[samcro1967]

No worries. Happy to help. Need testers to find things like this. Happy Holidays!

[leepeuker]

Alright, please try again with the new release 0.35.5

Edit: I have created a dedicated issue for the trakt api rate limit problem

[samcro1967]

I got to 2,262 this time. I think I have hit the API limit and not a new edge case. Is there something I can search for in the log file to confirm?

[leepeuker]

Yes, there should be an error log entry with the message Api error. Response status code: 429 at /app/src/Api/Trakt/TraktClient.php

[samcro1967]

I am not seeing that so maybe I have bumped into something else. Newest log attached from a clean run with a fresh container so the logs are not as long to sift through.

movary-2022-12-26T18-11-46.log

[leepeuker]

Hm. I cannot see any issues in the logs. What exactly is the problem? Is the job status Failed on the /job-queue page for the import?

[leepeuker]

Oh and please enable debug logging by setting LOG_LEVEL=debug when executing the import, this can provide more info

[samcro1967]

Plex reports 2,302 movies with 29 unwatched. So I am thinking I should end up with 2,273 movies. I am showing 2,264 in Movary so that leaves 9 missing. Trakt watch history shows 3,071 items, but also includes TV shows and movies I may have watched and then removed from Plex to free up storage. If you are not seeing any errors, guessing the discrepancy is probably movies I deleted. When I sync plex watched history to trakt, I don't believe it removes movies from the trakt watched history that have been removed from plex.

The job does show as completed successfully which it has never done before so it looks like it made it through everything successfully.

[leepeuker]

Okay, I will look into adding better debug logging for the import process, so that we can pin point which movies are missing to find out why. I will inform you when I have release these changes

[leepeuker]

So, with the latest realease the trakt import was a bit overhauled and the logging extend. When setting loglevel to debugin env, you should get information about how the app handled every trakt movie in the logs. It would improve the error search if we know which movies you are locally missing to check the logs for. Otherwise a fresh import on a new database should highlight possible errors during the import even better, if you got the time for this again (do not forget the rate limit issue, you probably have to restart it a few times again). I just imported my trakt library with 779 plays again and it imported exactly this number of plays.

[samcro1967]

I upgraded in place and did not pick up any more movies. I ran plex-trakt-sync manually. It is setup as a corn job with the container removed on exit so there are no container logs and the app has no log. I did find 6 movies when ran manually it could not match and I fixed them. Watched, I am now showing 2,271 in Movary, 2,276 in Plex, and 2,283 in Trakt. I see no errors in the Movary log. Interestingly enough plex-trakt-sync shows 2305 Movies in Plex where as plex shows 2306 (30 unwatched). So there is 1 orphan movie somewhere (maybe it is Annie lol). So Movary should be showing 2275 leaving a gap of 4 movies.

I think at this point the discrepancy is between what is watched in Plex and synced as watched in Trakt. I don't see an easy way to compare the 2. Trakt has an export feature, but I am not a VIP member. I'll see if I can figure out a way to do this and pin point the 4 missing movies as well as the additional movies in Trakt that are not in plex.

If you decide to implement a sync to Plex watched in the future, that may shed some additional light as well. If it would be helpful for me to start with a clean database and go through it again and provide you some logs, let me know. Happy to help if you need some additional testing and logs.

[samcro1967]

I was able to export Trakt watched history for movies only using this link. I converted the json to csv. I got Plex watched history from a Tautulli export in a csv. Was able to quickly compare them in Excel with a vlookup to find some oddities pretty quickly. Radarr and Trakt list a move as "3 Men and a Baby". Plex and Movarr list it as "Three Men and a Baby". Another movies is listed on Trakt as "13 Hours: The Secret Soldiers of Benghazi" and on Plex as "13 Hours" I think this is a deep rabbit hole not worth going any further. I am sure this will come down to data hygiene. 99.82% (2271/2275) is pretty good and not worth the hours of effort to identify the issue(s) with the data sources and then try to figure out how to resolve them programmatically. Appreciate your work. Pretty cool to be able to do some analysis on watched history. LMK if you need me to test anything else.