search

Taxel / PlexTraktSync

A python script that syncs the movies, shows and ratings between trakt and Plex (without needing a PlexPass or Trakt VIP subscription)
MIT License
1.64k stars 106 forks source link

Syncing watchlist between Trakt and Plex always adds back removed items #1247

Open rfgamaral opened 2 years ago

rfgamaral commented 2 years ago

Confirmation

The problem

Syncing my watchlist between Trakt and Plex does not preserve remove state, that is:

Error trace / logs

I removed 3 movies from my Trakt Watchlist and ran plextraktsync sync --sync=movies:

2022-12-01 12:19:58,946 INFO[PlexTraktSync]:PlexTraktSync [0.24.5]
2022-12-01 12:19:59,275 INFO[PlexTraktSync]:Sync Movie sections: [<PlexLibrarySection:movie:Concerts>, <PlexLibrarySection:movie:Movies>]
2022-12-01 12:20:01,890 INFO[PlexTraktSync]:Concerts processed in 2.6 seconds
2022-12-01 12:20:09,803 INFO[PlexTraktSync]:Movies processed in 7.9 seconds
2022-12-01 12:20:17,613 INFO[PlexTraktSync]:Adding 'Amsterdam' to Trakt watchlist
2022-12-01 12:20:31,097 INFO[PlexTraktSync]:Adding 'Black Panther: Wakanda Forever' to Trakt watchlist
2022-12-01 12:20:35,505 INFO[PlexTraktSync]:Adding 'Deepwater Horizon' to Trakt watchlist
2022-12-01 12:20:48,794 INFO[PlexTraktSync]:Skipping 'Untitled Ghostbusters: Afterlife Sequel' from Trakt watchlist because not found in Plex Discover
2022-12-01 12:20:52,170 INFO[PlexTraktSync]:Skipping 'The King' from Trakt watchlist because not found in Plex Discover
2022-12-01 12:20:52,185 INFO[PlexTraktSync]:Updated watchlist and/or liked list in 40.7 seconds
2022-12-01 12:20:52,322 INFO[PlexTraktSync]:Completed full sync in 53.1 seconds

I removed 3 movies from my Plext Watchlist and ran plextraktsync sync --sync=movies:

2022-12-01 12:21:57,098 INFO[PlexTraktSync]:PlexTraktSync [0.24.5]
2022-12-01 12:21:57,184 INFO[PlexTraktSync]:Sync Movie sections: [<PlexLibrarySection:movie:Concerts>, <PlexLibrarySection:movie:Movies>]
2022-12-01 12:21:58,890 INFO[PlexTraktSync]:Concerts processed in 1.7 seconds
2022-12-01 12:22:06,670 INFO[PlexTraktSync]:Movies processed in 7.8 seconds
2022-12-01 12:22:59,716 INFO[PlexTraktSync]:Skipping 'Untitled Ghostbusters: Afterlife Sequel' from Trakt watchlist because not found in Plex Discover
2022-12-01 12:23:04,296 INFO[PlexTraktSync]:Skipping 'The King' from Trakt watchlist because not found in Plex Discover
2022-12-01 12:23:04,919 INFO[PlexTraktSync]:Adding 'Amsterdam' to Plex watchlist
2022-12-01 12:23:06,450 INFO[PlexTraktSync]:Adding 'Black Panther: Wakanda Forever' to Plex watchlist
2022-12-01 12:23:07,845 INFO[PlexTraktSync]:Adding 'Deepwater Horizon' to Plex watchlist
2022-12-01 12:23:08,483 INFO[PlexTraktSync]:Updated watchlist and/or liked list in 1 min 0.1 seconds
2022-12-01 12:23:08,486 INFO[PlexTraktSync]:Completed full sync in 1 min 11.3 seconds

Expected behavior

Steps to reproduce the behavior

Any sync command results in the issue.

Inspect of problematic items

No response

Workarounds

No response

Install method

docker-compose

Config file contents

# Config File: /app/config/config.yml
cache:
  path: /app/config/.cache/trakt
config:
  dotenv_override: true
excluded-libraries:
- Samples
logging:
  append: true
  console_time: true
  debug: false
  filename: plextraktsync.log
  filter: null
plex:
  timeout: 30
sync:
  plex_to_trakt:
    collection: false
    ratings: false
    watched_status: true
    watchlist: true
  trakt_to_plex:
    liked_lists: false
    ratings: true
    watched_status: true
    watchlist: true
    watchlist_as_playlist: false
watch:
  add_collection: true
  remove_collection: true
  scrobble_threshold: 80
  username_filter: true
xbmc-providers:
  movies: imdb
  shows: tvdb

Version

0.24.5

Python Version

3.11.0

Operating System and Version

Synology (Linux)

simonc56 commented 2 years ago

When there is a Movie in one watchlist but not in the other watchlist, how could the script kow if it should remove it from the first watchlist or add it in the second wtchlist ?

rfgamaral commented 2 years ago

I can't work out the details right now because I don't know how exactly each API works, but it should be possible by keeping some local state for the previous sync for each watchlist, and do some condition checks.

Right now, the way it's implemented, it acts more like a "mirror" than a "sync".

glensc commented 2 years ago

FYI: After https://github.com/Taxel/PlexTraktSync/pull/1238 (0.24.6) watchlist sync is disabled if --sync=movies is specified.

simonc56 commented 2 years ago

FYI: After #1238 (0.24.6) watchlist sync is disabled if --sync=movies is specified.

Not relevant here because watchlist_as_playlist=false

rfgamaral commented 2 years ago

And I only used --sync=movies in my examples because I was testing and didn't want to go through syncing my TV Shows. For actual syncing, I use the Ofelia approach described in the README.

simonc56 commented 2 years ago

Workaround is to sync watchlist only one-way (plex_to_trakt or trakt_to_plex).

rfgamaral commented 2 years ago

Not really a workaround for me, I have no use for having just one of them enabled. Either 2-way sync works taking into account removed items, or I might as well just turn off both plex_to_trakt and trakt_to_plex 😔

glensc commented 1 year ago

I noticed the trakt response includes list: updated_at, could that be useful?

2022-12-27 09:39:21,429 DEBUG[PlexTraktSync]:Updated Trakt watchlist: {'deleted': {'movies': 3, 'shows': 1, 'seasons': 0, 'episodes': 0}, 'list': {'updated_at': '2022-12-27T07:39:20.000Z', 'item_count': 29}}

also, trakt side/sync/watchlist and /users/id/watchlist have listed_at timestamp. maybe useful?

    "listed_at": "2014-09-01T09:10:11.000Z",
glensc commented 1 year ago

so, the trakt response has:

{'id': 773647533,
  'listed_at': '2022-12-07T10:34:20.000Z',
  'movie': {'ids': {'imdb': 'tt0070948',
                    'slug': 'zardoz-1974',
                    'tmdb': 4923,
                    'trakt': 2858},
            'title': 'Zardoz',
            'year': 1974},
  'notes': None,
  'rank': 8,
  'type': 'movie'}

but all data that's not inside movie gets lost, because only movie data is collected:

so would need new data class to pytrakt that has listed_at, notes, movie, rank, type properties?

glensc commented 1 year ago

actually, watchlist_shows would pass the extra data to TVShow:

but I guess a new class with extra fields would be a cleaner solution.

simonc56 commented 1 year ago

I noticed the trakt response includes list: updated_at, could that be useful?

Explain how useful it is please.

glensc commented 1 year ago

I noticed the trakt response includes list: updated_at, could that be useful?

Explain how useful it is please.

please note the question form of the sentence.

the idea was to compare plex side watchlistedAt vs trakt side listed_at. for this to be useful, need the third timestamp, when was the last sync ran, such timestamp is not available.

do you see a solution how to do with only two timestamps? something like compare newest entry in plex = last update in plex side, and same for trakt?

rfgamaral commented 1 year ago

@glensc I have a Trakt Pro subscription, and I could make a feature request on the Trakt VIP forums to improve the API to allow this use case, I just need to know the exact requirements. Let me know how I can help.

simonc56 commented 1 year ago

the idea was to compare plex side watchlistedAt vs trakt side listed_at. for this to be useful, need the third timestamp, when was the last sync ran, such timestamp is not available.

Explain how comparing listedAt dates could help here. What would you do if plex_listed_at > trakt_listed_at for example ?

glensc commented 1 year ago

Explain how comparing listedAt dates could help here. What would you do if plex_listed_at > trakt_listed_at for example ?

You omitted the part where I said sync timestamp is needed.

in my mind, the algo would be:

  1. if item is missing in trakt watchlist, compare last_sync_date and plex_listed_at. if plex_listed_at > last_sync_date then add it to trakt otherwise drop from plex

the same for other direction.

glensc commented 1 year ago

yes. this could work


    def plex_watchlist_updated_at(self):
        from datetime import datetime, timezone
        from plexapi import utils
        updated_at = datetime(1970, 1, 1)
        for pm in self.plex_wl.values():
            listed_at = utils.toDatetime(pm._data.get("watchlistedAt"))
            updated_at = max(updated_at, listed_at)
        return updated_at.astimezone(timezone.utc)

    def trakt_watchlist_updated_at(self, watchlist):
        from datetime import datetime, timezone
        updated_at = datetime(1970, 1, 1).astimezone(timezone.utc)
        for tm in watchlist.values():
            listed_at = datetime.strptime(tm.listed_at, "%Y-%m-%dT%H:%M:%S.%f%z")
            updated_at = max(updated_at, listed_at)

        return updated_at

    def watchlist_sync_item(self, m: Media, dry_run=False):
        if not self.sync_wl:
            return

        plex_watchlist_updated_at = self.plex_watchlist_updated_at()
        print(plex_watchlist_updated_at)
        trakt_movies_watchlist_updated_at = self.trakt_watchlist_updated_at(self.trakt_wl_movies)
        print(trakt_movies_watchlist_updated_at)
        if plex_watchlist_updated_at > trakt_movies_watchlist_updated_at:
            print(f"Plex watchlist is newer")
        else:
            print(f"Trakt watchlist is newer")
glensc commented 1 year ago

extra thing to consider:

https://trakt.docs.apiary.io/#reference/sync/get-watchlist/get-watchlist:

Auto Removal

When an item is watched, it will be automatically removed from the watchlist. For shows and seasons, watching 1 episode will remove the entire show or season.

could maybe compare "played_on_trakt" state and make an extra decision.

simonc56 commented 1 year ago

You omitted the part where I said sync timestamp is needed.

How do you get it ?

        if plex_watchlist_updated_at > trakt_movies_watchlist_updated_at:
            print(f"Plex watchlist is newer")
        else:
            print(f"Trakt watchlist is newer")

I think those print() assertions are wrong because it forgets to read the "remove" actions. It only reads "add" actions on watchlists.

glensc commented 1 year ago

You omitted the part where I said sync timestamp is needed.

How do you get it ?

read last sentence:

        if plex_watchlist_updated_at > trakt_movies_watchlist_updated_at:
            print(f"Plex watchlist is newer")
        else:
            print(f"Trakt watchlist is newer")

I think those print() assertions are wrong because it forgets to read the "remove" actions. It only reads "add" actions on watchlists.

also must consider trakt removes automatically from Watchlist:

simonc56 commented 1 year ago

Removed automatically or manually from watchlist doesn't change the problem.

glensc commented 1 year ago

Perhaps one thing to do is not to add to watchlist items that are played.

glensc commented 1 year ago

found that trakt offers a response that includes when watchlist was last updated:

aleksandarmomic commented 1 year ago

This feature would be a great addition as I would like to use the watchlists in this way:

KoenBoone commented 1 year ago

This feature would be a great addition as I would like to use the watchlists in this way:

* Add a movie to Plex Watchlist (I expect it to be added to Trakt)

* I add a movie to Trakt Watchlist (I expect it to be added to Plex)

* I watch a movie on Plex and it detects the movie is watched and auto removes the movie from the Plex Watchlist (I expect that movie to also be removed from Trakt Watchlist)

* I watch a movie somewhere else and I manually mark the movie as watched in Trakt and I manually remove it from Trakt Watchlist (I expect that movie removed from Plex Watchlist also)

exactly, that is how sync should work, at the moment it's more 'tranfer' than 'sync' :-)

glensc commented 1 year ago

@KoenBoone opinions, etc trash should go to discussions. if you want to be helpful, provide solutions. your comment has been hidden.

KoenBoone commented 1 year ago

Well, i provided a solution, sadly you seem to think it is trash.

Best regards

Koen

Sent from my mobile

From: Elan Ruusamäe @.> Sent: Saturday, April 29, 2023 10:06:47 AM To: Taxel/PlexTraktSync @.> Cc: Koen Boone @.>; Mention @.> Subject: Re: [Taxel/PlexTraktSync] Syncing watchlist between Trakt and Plex always adds back removed items (Issue #1247)

@KoenBoonehttps://github.com/KoenBoone opinions, etc trash should go to discussionshttps://github.com/Taxel/PlexTraktSync/discussions. if you want to be helpful, provide solutions. your comment has been hidden.

— Reply to this email directly, view it on GitHubhttps://github.com/Taxel/PlexTraktSync/issues/1247#issuecomment-1528711913, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAHSV6XFJ7DBUXE7FYHP7X3XDTDZPANCNFSM6AAAAAASQXPRLM. You are receiving this because you were mentioned.Message ID: @.***>

glensc commented 1 year ago

exactly, that is how sync should work, at the moment it's more 'tranfer' than 'sync' :-)

you just commented some post, I read no solution from there.

KoenBoone commented 1 year ago

I stand corrected, i thought it was another issue (#1080), should have followed the link instead of replying by email :-)

Jpinilla712 commented 1 year ago

This feature would be a great addition as I would like to use the watchlists in this way:

  • Add a movie to Plex Watchlist (I expect it to be added to Trakt)
  • I add a movie to Trakt Watchlist (I expect it to be added to Plex)
  • I watch a movie on Plex and it detects the movie is watched and auto removes the movie from the Plex Watchlist (I expect that movie to also be removed from Trakt Watchlist)
  • I watch a movie somewhere else and I manually mark the movie as watched in Trakt and I manually remove it from Trakt Watchlist (I expect that movie removed from Plex Watchlist also)

I agree with this!

RileyXX commented 1 year ago

This feature would be a great addition as I would like to use the watchlists in this way:

  • Add a movie to Plex Watchlist (I expect it to be added to Trakt)
  • I add a movie to Trakt Watchlist (I expect it to be added to Plex)
  • I watch a movie on Plex and it detects the movie is watched and auto removes the movie from the Plex Watchlist (I expect that movie to also be removed from Trakt Watchlist)
  • I watch a movie somewhere else and I manually mark the movie as watched in Trakt and I manually remove it from Trakt Watchlist (I expect that movie removed from Plex Watchlist also)

Alternatively you could use a script specifically to remove watched items from plex/trakt watchlists.

Edit: After thinking about it this would be a great separate feature to add as an optional config setting. I opened a separate issue for it here https://github.com/Taxel/PlexTraktSync/issues/1494

Miladiir commented 5 months ago

Similar problem to synchronizing two filesystems, that do not know about each other. Take a look at how rclone solved with for the bisync feature: https://rclone.org/bisync/. Initial run: Get list from trakt and save the list with timestamp. Get list from plex and save the list with timestamp. Decide, which list to keep. Remove missing from plex? Remove missing from trakt? Keep both, as in, add missing items to the other list? Perform synchronisation action and save result (the two lists, with two timestamps)

Subsequent runs can then decide based on timestamp, which items where removed from each list or added to each list and replicate this action to the other list.

The only problem with this approach, is that the index needs to be protected. If the local copy of the list with the timestamps is tampered with, lost or in some other way corrupted, chaos can ensue. Worst case is, both lists get pruned of all items. Another failure case is that updates are lost, e.g. all removed items from last run are added back.

There is a lot of fun computer sciency topics to think about, e.g. https://en.wikipedia.org/wiki/Write%E2%80%93write_conflict. Don't think implementing this well would be trivial. Or maybe I am overthinking it.

glensc commented 5 months ago

Without adding local database conclusion made already earlier. Some attempts already exists:

Perhaps first step would be add plugin to fill the database using new plugin infrastructure:

RileyXX commented 5 months ago

If the local copy of the list with the timestamps is tampered with, lost or in some other way corrupted, chaos can ensue. Worst case is, both lists get pruned of all items. Another failure case is that updates are lost, e.g. all removed items from last run are added back.

I was thinking about ways you could implement this too and this was my conclusion as well. There is a lot of risk associated with trying to determine which items to delete. Trakt API goes down pretty frequently which can cause issues. Local database can be corrupted, deleted, etc. Sometimes Trakt API can also omit results when things are not working correctly (multiple pagination pages missing). I would assume the same risks for Plex API as well. There would need to be a lot of fail proof checks in order to accomplish this safely.

Some things I was considering in one of my projects (IMDB-Trakt-Syncer) that would apply here as well, schedule items to be removed. Periodically check all lists to verify the item should be deleted (to ensure no API errors), after a certain amount of checks or time passed then approve the scheduled items for deletion. Additionally, you could add a check so that if more than 50% of a list is to be deleted then stop the deletion (in order to prevent accidental deletion due to unforeseen issues). As you can see it would get very complicated and there is still risk of chaos ensuing as you mentioned.