Emby plugin syncing issues

[rudf0rd]

Hey not sure what information I can help with from the platform side, but I have been getting multiple requests per week to deduplicate histories. Duplicate plays are sent each time with the same watched_at timestamps, deduplication is possible, thankfully.

Please let me know how we can be of help.

[LukePulverenti]

Hi, can you please provide an example? Thanks.

[rudf0rd]

Sure. In what format would you like it?

[LukePulverenti]

A list of requests and times, ideally coupled with the corresponding emby server log.

[rudf0rd]

Ah, apologies.. I'm not an emby user myself. I'm a developer at Trakt.

[rudf0rd]

I can provide:

• a user with a locked account
• a breakdown of episodes/movies by:
  • unique media items
  • date

[techietravels]

embyserver-63796636800.txt

I was affected by the issue. I have attached a log from the day it looks my account got locked.

[rudf0rd]

And for the above user, here is a breakdown of how many episodes/movies were added per day

.

[LukePulverenti]

embyserver-63796636800.txt

I was affected by the issue. I have attached a log from the day it looks my account got locked.

Have you configured the scheduled task to sync from trakt on a schedule? That is possibly the reason for your issue as I see multiple executions of it in your log file. It does a full sync, not an incremental one. When you install the plugin it has no schedule. It's intended usage is more for initial import only.

[LePresidente]

So I just got this issue, what would the be the best approach then to sync to trakt on new watched shows if we can't schedule a task since it spams there api with a full sync?

[LukePulverenti]

So I just got this issue, what would the be the best approach then to sync to trakt on new watched shows if we can't schedule a task since it spams there api with a full sync?

Sync to or from trakt? Because syncing to trakt will be handled on the fly as you watch your content. You don't need to go back and run any task for that.

[LePresidente]

Sync to trakt, it's not clear from the plug in that it syncs on the fly and the task Is only for initial syncing. I think the scheduled task should be removed to be honest to stop this sort of issue or clearer description on what it is there for

[LukePulverenti]

I think the original developer made it a scheduled task out of convenience, because scheduled tasks can handle a lot of things without having to build them into the plugin, such as the UI for starting/stopping the task, monitoring progress, preventing simultaneous executions, etc.

[LePresidente]

Yea i do see that but there is no mention about the sync being on the fly. So if you look at the plugin at the moment it would be assumed that each schedule is required to sync to and from trakt. But that's not a emby issue just bad tooltip/documentation

[LukePulverenti]

I'm going to update the descriptions of the scheduled task to better illustrate their purposes.

[LePresidente]

Ok the above is still not working correctly. so doing a full sync seems to push how far in the episode you are (resume from) option to trakt.

I just cleared my library.db to try fix another issue and the trakt import restored all the old resume from times for old episodes. Marking them unwatched, watched removes the resume from option till the next trakt import which restores them again

[rudf0rd]

I'm still clearing 10+ accounts a week that have been duplicated over 200k items and had their accounts locked. Any progress on this issue? At a certain point, I will have to bounce the API key until the issue is dealt with.

[Wesleyl89]

I'm still clearing 10+ accounts a week that have been duplicated over 200k items and had their accounts locked. Any progress on this issue? At a certain point, I will have to bounce the API key until the issue is dealt with.

Yeah, that must pretty annoying. I think you just deduplicated my account (Ultimas), so for now I will just disable the plugin in Emby completely and keep this issue monitored.

[LukePulverenti]

I'm still clearing 10+ accounts a week that have been duplicated over 200k items and had their accounts locked. Any progress on this issue? At a certain point, I will have to bounce the API key until the issue is dealt with.

I think the problem is we have some users scheduling full syncs to run multiple times per day. We'll just have to update the plugin to now allow them to do this.

[Wesleyl89]

I'm still clearing 10+ accounts a week that have been duplicated over 200k items and had their accounts locked. Any progress on this issue? At a certain point, I will have to bounce the API key until the issue is dealt with.

I think the problem is we have some users scheduling full syncs to run multiple times per day. We'll just have to update the plugin to now allow them to do this.

That explains it, but could be confusing (it was for me). So basically you only run that once and then never again (unless everything is lost somehow).

As i'm reading the description now it says you only need to run it once, but maybe there should be an even more explicit warning that running it on a schedule can result in an account lock (and lots of duplicates in trakt).

[miniliQuid]

I'm still clearing 10+ accounts a week that have been duplicated over 200k items and had their accounts locked. Any progress on this issue? At a certain point, I will have to bounce the API key until the issue is dealt with.

I think the problem is we have some users scheduling full syncs to run multiple times per day. We'll just have to update the plugin to now allow them to do this.

That explains it, but could be confusing (it was for me). So basically you only run that once and then never again (unless everything is lost somehow).

As i'm reading the description now it says you only need to run it once, but maybe there should be an even more explicit warning that running it on a schedule can result in an account lock (and lots of duplicates in trakt).

Had this issue as well unknowingly. Seems like my lack of knowledge burned me on this one. Would be important to have a very clear warning or popup for that schedule task (or simply not allow it to be run on a schedule, only manually if possible.

Seems since I disabled the scheduled runs, this month looks clean from dupes, yet still checks of within trakt, what has been watched on emby.

[psxlover]

Because syncing to trakt will be handled on the fly as you watch your content. You don't need to go back and run any task for that.

@LukePulverenti I believe the scheduled tasks were created automatically when installing the plugin. Also the syncing done on the fly is not always possible. What about if you are not online when you watch something? It seems as if the plugin stopped being able to keep track what has been synced from a point on, so on each scheduled sync it synchronizes all watched media in the library.

I thought the issue was related to One Piece and it's big episode numbers since the first issue I had was that an episode wasn't marked as seen, before it started marking each other episode as seen multiple times. By the way none of the seen movies that I have in my library have duplicate entries, so perhaps only things seen after the issue was introduced are affected (there aren't many of them I usually delete seen content from the disk Emby sees). Although that wouldn't explain why others are getting so many duplicates per day.

deduplication is possible, thankfully.

@rudf0rd how? I've been removing duplicates manually till now :/

[rudf0rd]

I am manually clearing accounts with database queries.  Trying to get away from it because it’s kind of a time suck. On Dec 4, 2022 at 1:22 PM -0800, Alex Pap @.***>, wrote:

Because syncing to trakt will be handled on the fly as you watch your content. You don't need to go back and run any task for that. @LukePulverenti I believe the scheduled tasks were created automatically when installing the plugin. Also the syncing done on the fly is not always possible. What about if you are not online when you watch something? It seems as if the plugin stopped being able to keep track what has been synced from a point on, so on each scheduled sync it synchronizes all watched media in the library. I thought the issue was related to One Piece and it's big episode numbers since the first issue I had was that an episode wasn't marked as seen, before it started marking each other episode as seen multiple times. By the way none of the seen movies that I have in my library have duplicate entries, so perhaps only things seen after the issue was introduced are affected (there aren't many of them I usually delete seen content from the disk Emby sees). Although that wouldn't explain why others are getting so many duplicates per day. deduplication is possible, thankfully. @rudf0rd how? I've been removing duplicates manually till now :/ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[LukePulverenti]

Because syncing to trakt will be handled on the fly as you watch your content. You don't need to go back and run any task for that.

@LukePulverenti I believe the scheduled tasks were created automatically when installing the plugin. Also the syncing done on the fly is not always possible. What about if you are not online when you watch something? It seems as if the plugin stopped being able to keep track what has been synced from a point on, so on each scheduled sync it synchronizes all watched media in the library.

I thought the issue was related to One Piece and it's big episode numbers since the first issue I had was that an episode wasn't marked as seen, before it started marking each other episode as seen multiple times. By the way none of the seen movies that I have in my library have duplicate entries, so perhaps only things seen after the issue was introduced are affected (there aren't many of them I usually delete seen content from the disk Emby sees). Although that wouldn't explain why others are getting so many duplicates per day.

deduplication is possible, thankfully.

@rudf0rd how? I've been removing duplicates manually till now :/

There are two scheduled tasks registered by the plugin. The one that syncs from trakt is something that you could run on a regular basis if you also have other media software integrated with your trakt account. But that would be the only reason.

On the other hand, the task that syncs data from emby to trakt is not something that needs to be run repeatedly after initial setup.

[DM-AU]

Hi Luke,

Just adding what I have experienced recently. I imported a large amount of content to Emby, however the items were not marked as collected to Trakt. I had to manually run the full sync to get there status into Trakt.

[rudf0rd]

Hey @LukePulverenti I'm continuing to get spammed duplicate accounts every week from Emby. Would you guys consider adding some logic to your full sync task to only send watch history that hasn't already been added to their accounts?

[LukePulverenti]

Hey @LukePulverenti I'm continuing to get spammed duplicate accounts every week from Emby. Would you guys consider adding some logic to yo

Example? As mentioned above, you don't need to run the sync to trakt on a regular basis.

[rudf0rd]

But people are and I continue to get accounts I have to unlock every week.

[LukePulverenti]

But people are and I continue to get accounts I have to unlock every week.

That's due to the volume of requests, which you can bring down by not having the tasks run so frequently.

[psxlover]

I don't think frequency is the issue if he has a library with several thousand files.

[rudf0rd]

I apologize, I was unclear. I'm part of the Trakt team and I'm not asking for help using the plugin. I'm asking you if its possible to change the behavior so there aren't so many people getting their accounts locked due to the Emby plugin spamming our API.

[psxlover]

I also continue to get duplicate watched status for files I have watched and have not deleted from my disk. The tasks were there from the beginning, but at some point they started to sync episodes that had already been synced instead of just syncing newly watched episodes, so I now get two watch entries per day per watched episode.

[psxlover]

For example: On the ones I haven't deleted any entries, all but one entries have the same time. On episodes that I have cleared in the past all the new entries have the same time.

[bukkaroo]

I am having this issue as well. In fact, it's lead to my Trakt account getting locked twice. A bunch of watched items are ahowing up with 100s of dups with exactly the same start time. Maybe a simple dup check that does not pass any tiem that currently exists on Trakt with an identical start time? Happy to provide logs or other info as needed.

[LukePulverenti]

I am having this issue as well. In fact, it's lead to my Trakt account getting locked twice. A bunch of watched items are ahowing up with 100s of dups with exactly the same start time. Maybe a simple dup check that does not pass any tiem that currently exists on Trakt with an identical start time? Happy to provide logs or other info as needed.

What is the schedule for the two trakt scheduled tasks?

[psxlover]

@rudf0rd at least in my case it seems that the issue is in two anime series (although I don't have a big library so I can't say for sure if it is related to that).

Why do anime appear as [season]x[absolute number]([absolute number]) instead of [season]x[episode number]([absolute number]): In particular in One Piece after a point it's just [season]x[absolute number]

[rudf0rd]

@LukePulverenti at this point, does the scheduling really matter? This is a wide spread enough problem that I'm wasting time fixing this every week. Allowing a tool that can accidentally be configured to spam our API is getting extremely frustrating. Can you please address my previous question regarding adding logic to not send watch history already on Trakt?

[rudf0rd]

@psxlover asked internally. this was an artifact of switching from tvdb to tmdb. we're currently not worrying about updating one piece because we believe tmdb will be merging all of them into one season.

[psxlover]

@rudf0rd the same has happened in Naruto and Naruto Shippuden. Anyway that's of topic.

Is there anything common in the entries that get duplicated? My duplicates are One Piece and Naruto Shippuden episodes, both of which have only absolute numbering in Trakt.

Other series from my library, for example Danmachi, Re-Zero, Attack on Titan, Westworld and The Expanse don't have duplicate watch entries in trakt even though I have watched episodes in my library.

The only common thing I find between the two problematic entries in my library apart from the episode numbering in Trakt, is how long they have been part of my library.

[psxlover]

@LukePulverenti doubt this will help but here is a log from my server embyserver.txt. In the logs I manually executed the Sync library to trakt.tv task, and you can see that both times it says that it has 8 files to sync:

Info Trakt: Episodes to set watched: 8 Those 8 are the 4 watched One Piece and the 4 watched Naruto Shippuden episodes in my library. There are other series with watched episodes but only the ones from those two series have issues in my library.

Also you'll notice that both synchronizations fail. No idea if that has anything to do with the duplicate watched issue, probably not since the duplicates started earlier. Looking at the "Alerts" in the server dashboard all the synchronizations since 12/12/2022 have failed. First it was "ServiceUnavailable" (I'm guessing that was when trakt had database issues last year), then it switches to "Unauthorized" on the 20/12/2022 (there are no logs between the 15th and 20th) and starting from 20/01/2023 (I'm guessing that's when I changed trakt token) it changed to the "Bad Request" you see in the attached logs.

[bukkaroo]

I am having this issue as well. In fact, it's lead to my Trakt account getting locked twice. A bunch of watched items are ahowing up with 100s of dups with exactly the same start time. Maybe a simple dup check that does not pass any tiem that currently exists on Trakt with an identical start time? Happy to provide logs or other info as needed.

What is the schedule for the two trakt scheduled tasks?

Daily for each.

[LePresidente]

This plugin looks to be closed source or simply no one is maintaining it and in its current state, it's broken.

Full sync causes the above issue.

Scrobbler doesn't update trackt on when something is marked played/unplayed, so if you sync back to emby most of your episodes are back to half played when you did the full sync above.

So I think either remove this plugin since in its state is causing way more issues then anything else (I have actually disabled since my last post above). Or fix the scheduled task to work as people expect a full sync to trakt that actually has checks to not hammer the api, that works with the scrobbler.

Or fix the scobbler to do everything the sync does on all actions, and remove the scheduled task to sync

[miracle152005]

I am having the same issue. My Api access was revoked since this plugin spammed 200K play history items. Will someone be addressing it?

[rudf0rd]

Hey @LukePulverenti, back to bug you again. Please fix your plugin by end of next week or I'll be forced to revoke your API key. I just had 2 more weeks of locked accounts, one of which had over 8 million duplicates.

It really sucks to get all demanding. I would hope the Emby team would prefer to treat a community resource with respect rather than ignore a glaring issue that's been open since last August.

[LukePulverenti]

There should be some improvement already in the server beta channel. The sync to trakt task I think needs to be changed so that it's an incremental sync and not meant to do everything every single time.

For a quick fix in the meantime I can push an update out that adds a provision to both tasks so that they're only able to run every 24 hours at most, regardless of how the user has configured them.

Then once the sync has been reworked to be incremental, then that restriction can be removed.

[LukePulverenti]

An update went out yesterday to the Emby Trakt plugin. There should be sufficient throttling now to ensure that you stay within Trakt API limits. If you run into an issue please let me know and we'll make further changes. Thanks.

[rudf0rd]

Thank you for the movement on this issue. I'll continue to monitor on our side.

For clarity sake, can you confirm that you can no longer push a full sync of your history on a schedule? Is it only incremental updates now?

[LukePulverenti]

Thank you for the movement on this issue. I'll continue to monitor on our side.

For clarity sake, can you confirm that you can no longer push a full sync of your history on a schedule? Is it only incremental updates now?

By default it doesn't on a schedule, however the plugin gives users the opportunity to configure that. This is still in place and hasn't changed, however the update does reset the task schedules back to on-demand only because I think a lot of users may have set it up when it wasn't necessary.

The syncing of library content is already incremental in that it's only sending new additions and removals, but the watch data I think is where there's still room for improvement.

[douglasparker]

I get reports of incorrect watch data being duplicated on Trakt due to running "Export Library to Trakt" on a schedule.

@LukePulverenti while I understand you say that task shouldn't be ran on a schedule, I personally want it on a schedule because users can (and do) add their trakt setup at any time, and I'd like for them to get their history uploaded to trakt.

The proper way to deal with this is to perform an incremental sync and only sync what hasn't already been added to trakt.

@rudf0rd has been asking you to handle this for years, and for years you ignore his valid criticism with no regard to how this is affecting a free and public offering.

I've seen valid criticism blatantly ignored on the forums too. As a new Emby Premiere subscriber I am disappointed with the responses here.

[douglasparker]

This is affecting all of my users. Our Trakt data is now simply wrong and extremely inaccurate.

Here is an example of how my personal Trakt history is skewed:

[LukePulverenti]

@douglasparker it is an incremental sync. The scheduled task compares data with trakt and only sends the difference. Please reproduce the issue and then attach the emby server log. Thanks.