search

trakt / Plex-Trakt-Scrobbler

Add what you are watching on Plex to trakt.tv
1.45k stars 166 forks source link

Support for AniDB metadata agents #190

Closed fuzeman closed 8 years ago

fuzeman commented 9 years ago

We will need to integrate http://thexem.de to map AniDB IDs and season/episodes to the correct TheTvDB ID and season/episode structure.

Metadata Agents:

Benni-chan commented 9 years ago

problem with thexem: as far as I see, the api won't return any id, only binding. so if you are searching for a specific anidb-id, season, epnum, you get only a season and epnum for tvdb, but not the tvdb-id.

because of this (and maybe other limitations, i'm not sure anymore), I'm currently using this xml in my own script, where I need anidb/tvdb bindings: https://github.com/ScudLee/anime-lists/ the xml is not perfect, but works so far (has some errors in it, and the maintainer is not the fastest).

but I'm also currently developing an online database for these bindings, with (hopefully) easy editing/correcting posibilities and api-access. I'll post again, when I have something to show.

fuzeman commented 9 years ago

Strange, expected this data to be exposed in the xem API (it must be in their database).

Note: I'm currently redesigning the syncing system, so won't have time to work on this issue until the syncing changes have been completed.

M0UL commented 9 years ago

thexem support for mapings between the anidb end thtvdb would be nice.

copt3r commented 9 years ago

I think I got the same problem.

2015-06-19 14:13:02,907 (45c) : WARNING (logkit:19) - [plugin.sync.push] Ignored unmatched show "Xenosaga The Animation" [('hama', '2724')]>

Anyway I can change to tvdb? I tried fixing match, used tvdb, show is correct, loaded art work and whatnot but when running the sync again I got the same as above. I guess simply fixing the match with tvdb does not update this marker.

Edit: Fixing the match with TVDB scanner DID fix this "issue. It just had some delay.

Arno500 commented 9 years ago

Need this feature to use HAMA with Trakt.tv Scrobbler ! Can be enormous !

ZeroQI commented 9 years ago

Hama uses anidb ot tvdb and use as metadata.id "anidb-xxxx" or "tvdb-xxxxx". What can i change in the agent to make it compatible with Trakt or does it have to be done on Trakt side ?

fuzeman commented 9 years ago

@ZeroQI trakt doesn't support absolute numbering and AniDB ids, so episodes need to be mapped to the matching season + episode number and TVDB id.

There is a number of sources for AniDB -> TVDB mappings (xem, ScudLee/anime-lists, etc..), but they are either missing required data or updated very slowly.

The best solution would probably be a new web service as @Benni-chan described, not sure what the status of that project is though...

Arno500 commented 9 years ago

Sorry but this is the part I don't know. I think hama must be added in the Trakt code, and then the plugin must save TVDB id in the database and "pass" it to Trakt.

Arno500 commented 9 years ago

Sorry, didn't see Dean commentary. You're right. I've seen https://github.com/ScudLee/anime-lists who look quite updated. There is a web service for that, but i've forgotten the name, and he isn't very updated (very few, quite old animes) so not very useful at the moment.

KingJ commented 8 years ago

Where do things stand on this? xem seems to have most of the data for standard episodes, but things like OVAs, Specials etc are often missing. ScudLee's data seems to be more complete and he's also open to pull requests for any updates/additions that need to happen.

Even if a certain source doesn't have 100% coverage, it'd be good to move forward for now and then encourage people to update that source for any missing items.

Thanks for your efforts so far on this plugin!

fuzeman commented 8 years ago

I've started working on a mapping implementation that could apply to any metadata agent not just AniDB, ScudLee/anime-lists will be our primary data source of mappings for AniDB though.

Can't give any ETA as it's still very early in development (started this week). I'll update this issue when a beta release is ready, so subscribe to this issue if you want a notification on updates :smiley:

KingJ commented 8 years ago

That's great to hear! I'm more than happy to help test, so i've subscribed in order to keep informed about any updates.

fuzeman commented 8 years ago

AniDB scrobbling is now ready for testing on the feature/oem.zip branch.

Notes:

KingJ commented 8 years ago

Thanks for releasing this test version!

I've loaded it on to my testing instance of Plex and tried to scrobble a few shows that were added using @ZeroQI 's Scanner and Metadata Agent. For the most part, scrobbling seemed to work as expected. I tested a variety of content including;

The one thing that didn't match correctly is Spice and Wolf's (aid 5406) special episode ("Wolf and a Tail of Happiness", AniDB lists it as special 1 and so does TVDB). This instead got matched to Season 1 Episode 1 ("Wolf and Best Clothes"). However, I think this is an issue with the list data rather than the plugin itself due to the following log message;

2016-05-10 19:36:14,578 - oem.media.show.mapper (500c) : WARNING (oem.media.show.mapper:27) - Unable to find season 0 in show <Show tvdb: '81178', names: set(['Ookami to Koushinryou'])>

I'll test a few more things this evening and report back how it goes. Thanks again for your work on this!

KingJ commented 8 years ago

Some more tests on things that often involve different seasons/specials mapping, these all mapped correctly;

Didn't map correctly;

I had a look at ScudLee/anime-lists for both of these and it looks like there's no mapping for them, so it's a data source issue instead of a plugin mapping issue. Here's the current data;

`

Clannad 
<mapping-list>
  <mapping anidbseason="0" tvdbseason="1">;1-23;</mapping>
</mapping-list>
<before>;1-23;</before>

`

`

Clannad: After Story 
<mapping-list>
  <mapping anidbseason="0" tvdbseason="2">;1-23;2-24;</mapping>
</mapping-list>

</anime`

I've gone ahead and submitted PR #7 on ScudLee/anime-lists to correct it with what I think is the correct mapping and format. Out of interest once this gets merged, how does this get updated in your OpenEntityMap/oem-database-anidb-tvdb repo?

fuzeman commented 8 years ago

@KingJ Thanks for your further testing, there is definitely some missing/invalid data in https://github.com/ScudLee/anime-lists but sadly it appears changes aren't being merged in currently (no activity since March).

I'm considering starting a fork to get changes applied right now, and will hopefully have the commits merged into https://github.com/ScudLee/anime-lists in the future. (will post a comment here if this happens)

Out of interest once this gets merged, how does this get updated in your OpenEntityMap/oem-database-anidb-tvdb repo?

I've got an automatic importer that just parses the XML files and updates the OEM databases with any additions/changes. (this will be open-source soon, along with all the other OEM libraries)

Planning on having this setup to poll https://github.com/ScudLee/anime-lists for changes and post PRs to the database repositories automatically, but will currently just run the importer + commit changes to the repositories manually.

fuzeman commented 8 years ago

I've forked anime-lists to https://github.com/fuzeman/anime-lists, feel free to post issues and pull requests to that repository if you see any AniDB mapping issues.

@KingJ Looking at your PR (https://github.com/ScudLee/anime-lists/pull/71), there actually shouldn't be any need to add that mapping as the season and episode numbers aren't being changed between AniDB and TVDb.

I'm guessing something else is causing the mapping issue then.. will check what I can find in the OEM client.

fuzeman commented 8 years ago

I've fixed a few issues with AniDB episode mappings, download the beta branch for the latest changes.

KingJ commented 8 years ago

I've updated to the beta version on my test instance. Happy to report that all of the episodes I reported as not mapping correctly previously are now scrobbled correctly to Trakt with this version. I didn't find any new ones that didn't map correctly either.

I also gave movie scrobbling a try, everything I tried worked without issue.

Did you work out what my mapping PR should have been? I'm curious to know how it should be in case I come across any similar mis-mappings in the future and want to correct them.

Thanks for your work on this!

fuzeman commented 8 years ago

That's great, will be starting work on AniDb syncing today :smile:

Did you work out what my mapping PR should have been? I'm curious to know how it should be in case I come across any similar mis-mappings in the future and want to correct them.

Your PR was just a mapping for S00E02 -> S00E02 and S00E03 -> S00E03, there is no need to define these in anime-lists as the plugin will fallback to that if no mappings exist anyway.

KingJ commented 8 years ago

Very excited for syncing support! I can see you've made some commits to the feature/oem-syncing branch, is it in a ready state for me to download and test or should I hold off for the moment?

I think I understand the mapping syntax now, and why my PR was ultimately futile :) . Presumably the beta version from last week had some fixes that caused those episodes to map correctly?

fuzeman commented 8 years ago

Very excited for syncing support! I can see you've made some commits to the feature/oem-syncing branch, is it in a ready state for me to download and test or should I hold off for the moment?

Just finished up syncing support, it's available on the beta branch now.

I think I understand the mapping syntax now, and why my PR was ultimately futile :) . Presumably the beta version from last week had some fixes that caused those episodes to map correctly?

Yeah, the initial implementation wasn't mapping those episodes correctly.

KingJ commented 8 years ago

I've just given syncing support a quick try, but when running a pull from Trakt it encountered an error. The stack trace is;

2016-05-26 23:09:03,216 - plugin.sync.modes.core.base.mode (4fdc) :  DEBUG (plugin.sync.modes.core.base.mode:238) - [anidb/7439] - Mapped to: <Movie imdb: ['tt1754177', 'tt2231375'], names: set(['Mardock Scramble'])>
2016-05-26 23:09:03,217 - plugin.sync.main                 (4fdc) :  WARNING (plugin.sync.main:163) - Exception raised in run(): unhashable type: 'list'
Traceback (most recent call last):
  File "C:\Users\Username\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\main.py", line 159, in run_wrapper
    self.run()
  File "C:\Users\Username\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\main.py", line 245, in run
    'stop'
  File "C:\Users\Username\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\main.py", line 262, in _trigger
    func()
  File "C:\Users\Username\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\elapsed\main.py", line 64, in inner
    return func(*args, **kwargs)
  File "C:\Users\Username\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\modes\pull\__init__.py", line 47, in run
    self.execute_children('run')
  File "C:\Users\Username\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\modes\core\base\mode.py", line 136, in execute_children
    func()
  File "C:\Users\Username\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\elapsed\main.py", line 64, in inner
    return func(*args, **kwargs)
  File "C:\Users\Username\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\modes\pull\shows.py", line 69, in run
    pk = self.trakt.table('shows').get(key)
TypeError: unhashable type: 'list'

The line immediately before the stack trace concerns Mardock Scramble, which I have in my Anime Series library. However, it looks like it was (correctly) matched to a movie but in this case, the movie is listed as 3 separate episodes on AniDB.

Before the crash, it correctly synced the viewstates for most of my existing anime films (hooray!). The anime series library wasn't synced but i'm guessing that's because the pull terminated early when that error occured.

fuzeman commented 8 years ago

@KingJ Will have a look later today, mapping to multiple movies ['tt1754177', 'tt2231375'] is likely causing this issue.

fuzeman commented 8 years ago

@KingJ That issue should be fixed on the beta branch now.

Items in anime-lists with multiple keys (imdbid="tt1754177,tt2231375") will just be ignored for now, need to do some work on the OEM importer to get these matching correctly.

KingJ commented 8 years ago

@fuzeman Thanks for the quick turnaround! I've just updated and run a pull and everything appears to have synced down from Trakt!

Searching through the log for ERROR or WARNING I found the following entries;

2016-05-30 14:35:40,911 - plugin.sync.modes.core.base.mode (22f8) :  DEBUG (plugin.sync.modes.core.base.mode:282) - [anidb/513] (S01E01) - Mapped to: <MovieMatch (imdb: 'tt0301082')>
2016-05-30 14:35:40,911 - plugin.sync.main                 (22f8) :  WARNING (plugin.sync.main:163) - Exception raised in run(): 'MovieMatch' object has no attribute 'season_num'
Traceback (most recent call last):
  File "C:\Users\Kingsley\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\main.py", line 159, in run_wrapper
    self.run()
  File "C:\Users\Kingsley\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\main.py", line 245, in run
    'stop'
  File "C:\Users\Kingsley\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\main.py", line 262, in _trigger
    func()
  File "C:\Users\Kingsley\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\elapsed\main.py", line 64, in inner
    return func(*args, **kwargs)
  File "C:\Users\Kingsley\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\modes\pull\__init__.py", line 47, in run
    self.execute_children('run')
  File "C:\Users\Kingsley\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\modes\core\base\mode.py", line 137, in execute_children
    func()
  File "C:\Users\Kingsley\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\elapsed\main.py", line 64, in inner
    return func(*args, **kwargs)
  File "C:\Users\Kingsley\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\modes\pull\shows.py", line 94, in run
    supported, p_guid, season_num, episode_num = self.process_guid_episode(p_guid, season_num, episode_num)
  File "C:\Users\Kingsley\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\elapsed\main.py", line 64, in inner
    return func(*args, **kwargs)
  File "C:\Users\Kingsley\AppData\Local\Plex Media Server\Plug-ins\Trakttv.bundle\Contents\Libraries\Shared\plugin\sync\modes\core\base\mode.py", line 285, in process_guid_episode
    return True, Guid.construct(service, key), match.season_num, match.episode_num
AttributeError: 'MovieMatch' object has no attribute 'season_num'

This appears to be due to a metadata error on my end - the Cardcaptor Sakura series (AniDB ID 102) has instead been recognised by Hama as the movie (AniDB ID 513). After fixing the mapping and re-running the pull, 

2016-05-30 14:35:40,045 - oem.services.anidb               (22f8) :  WARNING (oem.services.anidb:67) - [anidb/513] Unable to find mapping for <EpisodeIdentifier S00E04>

Related to the previous error - same anidb ID.

2016-05-30 14:36:34,961 - root                             (4630) :  INFO (root:265) - Exception in I/O handler for fd 1400
Traceback (most recent call last):
  File "C:\Program Files (x86)\Plex\Plex Media Server\Resources\Plug-ins-5089475\Framework.bundle\Contents\Resources\Platforms\Shared\Libraries\tornado\ioloop.py", line 256, in start
    self._handlers[fd](fd, events)
  File "C:\Program Files (x86)\Plex\Plex Media Server\Resources\Plug-ins-5089475\Framework.bundle\Contents\Resources\Platforms\Shared\Libraries\tornado\stack_context.py", line 128, in wrapped
    callback(*args, **kwargs)
  File "C:\Program Files (x86)\Plex\Plex Media Server\Resources\Plug-ins-5089475\Framework.bundle\Contents\Resources\Platforms\Shared\Libraries\tornado\httpserver.py", line 219, in _handle_events
    connection, address = self._socket.accept()
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 10035] A non-blocking socket operation could not be completed immediately
2016-05-30 14:36:35,017 - root                             (4630) :  INFO (root:265) - Exception in I/O handler for fd 1400
Traceback (most recent call last):
  File "C:\Program Files (x86)\Plex\Plex Media Server\Resources\Plug-ins-5089475\Framework.bundle\Contents\Resources\Platforms\Shared\Libraries\tornado\ioloop.py", line 256, in start
    self._handlers[fd](fd, events)
  File "C:\Program Files (x86)\Plex\Plex Media Server\Resources\Plug-ins-5089475\Framework.bundle\Contents\Resources\Platforms\Shared\Libraries\tornado\stack_context.py", line 128, in wrapped
    callback(*args, **kwargs)
  File "C:\Program Files (x86)\Plex\Plex Media Server\Resources\Plug-ins-5089475\Framework.bundle\Contents\Resources\Platforms\Shared\Libraries\tornado\httpserver.py", line 219, in _handle_events
    connection, address = self._socket.accept()
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 10035] A non-blocking socket operation could not be completed immediately

Not sure what caused these. The surrounding entries didn't seem to give any hints.

After fixing the Cardcaptor Sakura mapping, I restarted Plex and ran another pull. No errors this time, but a few warnings were logged;

2016-05-30 14:53:17,278 - plugin.sync.modes.pull.movies    (4814) :  INFO (plugin.sync.modes.pull.movies:27) - Found 1 unsupported movie(s)
    [          None] Service not supported (4 items)
2016-05-30 14:53:17,278 - plugin.sync.modes.pull.movies    (4814) :  WARNING (plugin.sync.modes.pull.movies:34) - Unsupported service: None

I'm guessing this is related to the ignore fix you added?

2016-05-30 14:53:19,756 - plugin.modules.mapper.main       (4814) :  WARNING (plugin.modules.mapper.main:125) - Unable to find item for anidb: 10563

A missing Anime Lists mapping - anidb 10563 S01E01, 02, 03, 04 should match to tvdb 76703 S0E50, 55, 56, 59.

2016-05-30 14:53:19,766 - plugin.modules.mapper.main       (4814) :  WARNING (plugin.modules.mapper.main:125) - Unable to find item for anidb: 6262

A missing Anime Lists mapping, but can be safely ignored. This is a music video (episode T2 on that anidb ID) that somehow found its way in to my anime collection.

2016-05-30 14:53:20,454 - plugin.sync.modes.core.base.mode (4814) :  INFO (plugin.sync.modes.core.base.mode:279) - [anidb/7439] - Unsupported key: ['tt1754177', 'tt2231375']
2016-05-30 14:53:20,457 - oem.services.anidb               (4814) :  WARNING (oem.services.anidb:67) - [anidb/7439] Unable to find mapping for <EpisodeIdentifier S01E02>

The Mardock Scramble multiple key issue again.

2016-05-30 14:53:21,262 - oem.services.anidb               (4814) :  WARNING (oem.services.anidb:67) - [anidb/1543] Unable to find mapping for <EpisodeIdentifier S01E01>
2016-05-30 14:53:21,263 - plugin.core.logger.handlers.error_reporter.ErrorReporter (4814) :  DEBUG (plugin.core.logger.handlers.error_reporter.ErrorReporter:610) - Sending message of length 948 to threaded+requests+https://sentry.skipthe.net/api/1/store/
2016-05-30 14:53:21,263 - plugin.sync.modes.core.base.mode (4814) :  DEBUG (plugin.sync.modes.core.base.mode:287) - Unable to find mapping for <Guid - service: 'anidb', id: 1543> S01E01
2016-05-30 14:53:21,265 - oem.services.anidb               (4814) :  WARNING (oem.services.anidb:67) - [anidb/1543] Unable to find mapping for <EpisodeIdentifier S01E02>
2016-05-30 14:53:21,266 - plugin.core.logger.handlers.error_reporter.ErrorReporter (4814) :  DEBUG (plugin.core.logger.handlers.error_reporter.ErrorReporter:610) - Sending message of length 952 to threaded+requests+https://sentry.skipthe.net/api/1/store/
2016-05-30 14:53:21,266 - plugin.sync.modes.core.base.mode (4814) :  DEBUG (plugin.sync.modes.core.base.mode:287) - Unable to find mapping for <Guid - service: 'anidb', id: 1543> S01E02
2016-05-30 14:53:21,267 - oem.services.anidb               (4814) :  WARNING (oem.services.anidb:67) - [anidb/1543] Unable to find mapping for <EpisodeIdentifier S01E03>
2016-05-30 14:53:21,269 - plugin.core.logger.handlers.error_reporter.ErrorReporter (4814) :  DEBUG (plugin.core.logger.handlers.error_reporter.ErrorReporter:610) - Sending message of length 948 to threaded+requests+https://sentry.skipthe.net/api/1/store/

This continues all the way up to E26. Seems to be due to a missing Anime Lists entry, AniDB ID 1543 should map to TVDB ID 79089. I'm actually kind of surprised this mapping is missing given the historical popularity of this series!

2016-05-30 14:53:21,496 - plugin.sync.modes.core.base.mode (4814) :  DEBUG (plugin.sync.modes.core.base.mode:287) - Unable to find mapping for <Guid - service: 'anidb', id: 516> S01E01
2016-05-30 14:53:21,496 - oem.services.anidb               (4814) :  WARNING (oem.services.anidb:67) - [anidb/516] Unable to find mapping for <EpisodeIdentifier S01E02>
2016-05-30 14:53:21,497 - plugin.core.logger.handlers.error_reporter.ErrorReporter (4814) :  DEBUG (plugin.core.logger.handlers.error_reporter.ErrorReporter:610) - Sending message of length 948 to threaded+requests+https://sentry.skipthe.net/api/1/store/
2016-05-30 14:53:21,499 - plugin.sync.modes.core.base.mode (4814) :  DEBUG (plugin.sync.modes.core.base.mode:287) - Unable to find mapping for <Guid - service: 'anidb', id: 516> S01E02
2016-05-30 14:53:21,500 - oem.services.anidb               (4814) :  WARNING (oem.services.anidb:67) - [anidb/516] Unable to find mapping for <EpisodeIdentifier S01E03>
2016-05-30 14:53:21,500 - plugin.core.logger.handlers.error_reporter.ErrorReporter (4814) :  DEBUG (plugin.core.logger.handlers.error_reporter.ErrorReporter:610) - Sending message of length 948 to threaded+requests+https://sentry.skipthe.net/api/1/store/

This continues all the way up to EP38. Another missing Anime Lists mapping, AniDB ID 516 should map to TVDB ID 76045.

2016-05-30 14:53:21,493 - plugin.sync.modes.core.base.mode (4814) :  DEBUG (plugin.sync.modes.core.base.mode:287) - Unable to find mapping for <Guid - service: 'anidb', id: 516> S00E201
2016-05-30 14:53:21,494 - oem.services.anidb               (4814) :  WARNING (oem.services.anidb:67) - [anidb/516] Unable to find mapping for <EpisodeIdentifier S01E01>
2016-05-30 14:53:21,496 - plugin.core.logger.handlers.error_reporter.ErrorReporter (4814) :  DEBUG (plugin.core.logger.handlers.error_reporter.ErrorReporter:610) - Sending message of length 948 to threaded+requests+https://sentry.skipthe.net/api/1/store/
2016-05-30 14:53:21,496 - plugin.sync.modes.core.base.mode (4814) :  DEBUG (plugin.sync.modes.core.base.mode:287) - Unable to find mapping for <Guid - service: 'anidb', id: 516> S01E01
2016-05-30 14:53:21,496 - oem.services.anidb               (4814) :  WARNING (oem.services.anidb:67) - [anidb/516] Unable to find mapping for <EpisodeIdentifier S01E02>
2016-05-30 14:53:21,497 - plugin.core.logger.handlers.error_reporter.ErrorReporter (4814) :  DEBUG (plugin.core.logger.handlers.error_reporter.ErrorReporter:610) - Sending message of length 948 to threaded+requests+https://sentry.skipthe.net/api/1/store/
2016-05-30 14:53:21,499 - plugin.sync.modes.core.base.mode (4814) :  DEBUG (plugin.sync.modes.core.base.mode:287) - Unable to find mapping for <Guid - service: 'anidb', id: 516> S01E02
2016-05-30 14:53:21,500 - oem.services.anidb               (4814) :  WARNING (oem.services.anidb:67) - [anidb/516] Unable to find mapping for <EpisodeIdentifier S01E03>
2016-05-30 14:53:21,500 - plugin.core.logger.handlers.error_reporter.ErrorReporter (4814) :  DEBUG (plugin.core.logger.handlers.error_reporter.ErrorReporter:610) - Sending message of length 948 to threaded+requests+https://sentry.skipthe.net/api/1/store/

Cardcaptor Sakura strikes again with a missing Anime List mapping. AniDB ID 516 should map to TVDB ID 70668.

That's all the warnings. Aside from a few mapping issues everything appears to have worked OK! I also tried making an unwatched episode as watched and then triggering a sync, it appeared in my Trakt history as a watch as expected.

Many thanks for your work on this as always! :)

fuzeman commented 8 years ago

Just pushed a bunch of updates to the beta branch, and published an update for the anidb -> tvdb and anidb -> imdb mapping databases.

A missing Anime Lists mapping - anidb 10563 S01E01, 02, 03, 04 should match to tvdb 76703 S0E50, 55, 56, 59.

Just added this mapping in v1.16.1.

This continues all the way up to E26. Seems to be due to a missing Anime Lists entry, AniDB ID 1543 should map to TVDB ID 79089. I'm actually kind of surprised this mapping is missing given the historical popularity of this series!

We have this in the database here, but as it's defined with "default_season": "a" it isn't supported yet.

I've got some changes planned to the database updater to support these mappings without any changes required on the Trakt.tv API. Will hopefully have this ready sometime this month.

This continues all the way up to EP38. Another missing Anime Lists mapping, AniDB ID 516 should map to TVDB ID 76045.

Same as above, it's in the database here but isn't supported yet.

KingJ commented 8 years ago

I just updated to the latest beta and ran another full sync. No errors, but there was one single warning;

2016-06-05 15:39:56,088 - plugin.sync.modes.push.shows (3d80) : INFO (plugin.sync.modes.push.shows:27) - Found 1 unsupported show(s) [ anidb] Service not supported (142 items) 2016-06-05 15:39:56,088 - plugin.sync.modes.push.shows (3d80) : WARNING (plugin.sync.modes.push.shows:41) - Unsupported service: anidb

This log messages is immediately before the new "Unable to find ..." log messages you added in 1bf40b8e562ac5d49671ca6c7d0565be412a11b2 (which is a really nice improvement by the way, much easier to see any mapping errors!). These are all for shows that I either have on Trakt, but not on this Plex instance or shows mentioned previously that don't match properly.

fuzeman commented 8 years ago

The beta branch has been updated with changes from the latest stable release, these changes should fix a plugin crash that started with v0.9.17.* of Plex Media Server.

KingJ commented 8 years ago

@fuzeman I loaded this up on to my test Plex instance, no issues whatsoever. I then upgraded from the 1.0.2.1 release to the beta version for the first time on my "regular" Plex instance and kicked off a sync/pull. All of the existing watches on Trakt came down successfully and all of the things i've been watching over the last few months since installing Plex synced back up to Trakt along with their respective watch dates. Haven't noticed any issues in the logs so far (beyond things mentioned previously) and everything's looking good on Plex and Trakt too! :smile:

fuzeman commented 8 years ago

Just pushed an update to the beta branch and AniDB mapping database to support shows that had absolute mappings (they have been converted to season + episode mappings).

I'm not sure which previously reported issues will be fixed by this update, so please report back with any mapping errors you find with this update (even if they have been previously reported here).

The library built for updating + retrieving AniDB mappings has been open-sourced, and is now available at https://github.com/OpenEntityMap

No documentation for OEM has been written yet, send me an email if you want some help integrating this into other projects :smiley:

KingJ commented 8 years ago

@fuzeman Updated this on my primary Plex instance and then kicked off a sync followed by a pull. Series that had issues previously due to using absolute numbering (Cardcaptor Sakura and Princess Tutu) now sync correctly both ways :smile: . That said, the "Unable to find x show(s) in Plex" lists Cardcaptor Sakura's TVDB ID (70668) and another that doesn't seem to be on there at all (273656).

In the "Unable to find 7 episode(s) in Plex" there's a few oddities too, for example;

('tvdb', '114921') (2, 25)

114921 is a Certain Scientific Railgun, but there is no Episode 25?

('tvdb', '273656') (1, 2) (1, 3) (1, 1) (1, 4)

ID 273656 again which doesn't seem to exist in the TVDB.

The log does have a few errors and tracebacks in it related to an unsupported locale setting and some Tornado errors. It looks like these have all been sent to Sentry though, let me know if you need the raw logs.

It's not being logged as an error, but i'm also seeing the following message in Trakt's logs roughly every minute; (7f2305ffb700) : INFO (plex_activity.sources.s_logging.main:84) - Unable to read log file

In general though, happy to report that absolute mappings seem to be working ok!

fuzeman commented 8 years ago

@KingJ

That said, the "Unable to find x show(s) in Plex" lists Cardcaptor Sakura's TVDB ID (70668) and another that doesn't seem to be on there at all (273656).

Will need to check why 70668 is listed, possibly just a bug in the reporting of missing shows.

273656 is still listed on trakt.tv as https://trakt.tv/shows/nekomonogatari-black-tsubasa-family, but has since been deleted from TheTVDB as it was a duplicate (details).

('tvdb', '114921') (2, 25)

114921 is a Certain Scientific Railgun, but there is no Episode 25?

Trakt.tv has S02E25 listed for some reason https://trakt.tv/shows/2009-30843, so that would explain this message. I've triggered a data refresh on Trakt.tv, so this issue might resolve itself.

('tvdb', '273656') (1, 2) (1, 3) (1, 1) (1, 4)

ID 273656 again which doesn't seem to exist in the TVDB.

I'm guessing you've marked this as watched on Trakt.tv, but it's since been deleted from TheTVDB.

The log does have a few errors and tracebacks in it related to an unsupported locale setting and some Tornado errors. It looks like these have all been sent to Sentry though, let me know if you need the raw logs.

Yeah, these errors/warnings should be safe to ignore. The tornado errors are from the Plex Plugin Framework.

It's not being logged as an error, but i'm also seeing the following message in Trakt's logs roughly every minute; (7f2305ffb700) : INFO (plex_activity.sources.s_logging.main:84) - Unable to read log file

Not sure what would cause this, you could probably silence these messages by changing "Activity mode" to "WebSocket" (via Plex/Web).

In general though, happy to report that absolute mappings seem to be working ok!

Great, getting closer to a stable release now. :smile:

KingJ commented 8 years ago

@fuzeman Looks like you were right on Nekomonogatari. I unmarked Trakt's 273656 version and marked the respective specials in 102261 as watched instead. After a sync, the episodes were marked as watched in Plex - hurrah!

I changed the Activity Mode in the plugin settings from Automatic to WebSocket as suggested, that appears to have fixed the "Unable to read log file" errors.

In the Trakt channel menu, I occasionally have an exception listed of "[Exception] 'NoneType' object has no attribute 'close'". Searching through my log archive, the only time this appears in the logs is as part of a Tornado error trace, so i'm guessing that's safe to ignore?

Great to hear the plugin is close to a stable release, exciting times! Is there anything else you want me to test specifically before release? At the moment, i'm just using it on my main Plex server on a day-to-day basis and noting any issues as they crop up.

fuzeman commented 8 years ago

In the Trakt channel menu, I occasionally have an exception listed of "[Exception] 'NoneType' object has no attribute 'close'". Searching through my log archive, the only time this appears in the logs is as part of a Tornado error trace, so i'm guessing that's safe to ignore?

Hmm, these errors shouldn't be displayed in the channel menu.. will make sure they are filtered out in a future update.

Great to hear the plugin is close to a stable release, exciting times! Is there anything else you want me to test specifically before release? At the moment, i'm just using it on my main Plex server on a day-to-day basis and noting any issues as they crop up.

Can't think of anything specific to test for, just want to ensure movies + episodes are mapped correctly.

The only feature remaining to implement is the mapping of multi-part episodes, where multiple episodes on AniDB map to one episode on TheTVDB (and vice versa). (will possibly have this implemented later this week)

iTorrek commented 8 years ago

Am I right that this new update shall provide nice work between TRAKT + ASS and HAMA? Then its not working. Please, look the attached file. Trakt version is 1.1.0.4-beta

com.plexapp.plugins.trakttv.txt

fuzeman commented 8 years ago

@iTorrek

Am I right that this new update shall provide nice work between TRAKT + ASS and HAMA?

Yes, Absolute Series Scanner + HAMA is currently supported on the beta branch.

Then its not working. Please, look the attached file.

I can't see any obvious issues in that log file.

Could you provide another log file (after attempting another sync) with all the plugin "Logging" options set to "DEBUG" (via Plex/Web)? debug messages will provide a lot more detail on what is happening during the syncing process.

iTorrek commented 8 years ago

Did it, please, take a look com.plexapp.plugins.trakttv.txt

fuzeman commented 8 years ago

Just pushed a big update to the beta branch, let me know if you find any issues :smile: (release notes)

@iTorrek Doesn't look like any items (with AniDB metadata) are being detected, could you check what the "guid" of your items are by doing the following steps:

  1. Open an episode or movie in Plex/Web (from your AniDB library)
  2. Click the three horizontal dots (on the left)
  3. Click "Info"
  4. Click "View XML"
  5. Either find the guid="..." value and paste it here, or download and attach the entire file here
KingJ commented 8 years ago

@fuzeman Just updated to the latest beta and tested it out (sorry for the delay, been away for a few days!). No errors during a sync, but there are a few match issues. I really like the new report feature, makes it easier to find out what didn't map :smile: . The Tornado errors have disappeared from the channel UI.

The following movies didn't match;

In shows, Cardcaptor Sakura (70668) still doesn't match. Other than that, all good :smile: .

In episodes, seems all ok!

I still need to test a few other things, but otherwise it's all looking good :smile: . I was running the previous beta (.2 I think?) for other a month, and things have been scrobbling/syncing nicely across a multitude of Plex devices. Thanks as always for your work on this.

fuzeman commented 8 years ago

@KingJ

The following movies didn't match;

imdb/tt0190641- in my anime movies library. imdb/tt3354096 and imdb/tt3354254 - in my anime series library.

I've just pushed an update to add/fix the mappings for those movies, let me know if any of them are still broken.

Cardcaptor Sakura (70668) still doesn't match

Can't see any obvious issues with Cardcaptor Sakura, will investigate what's happening with that show later today.

Note: I've also pushed an update to the beta branch, this fixes an issue scrobbling mapped episodes (ZeroDivisionError exception).

KingJ commented 8 years ago

I updated to the latest beta and started the PMS process again, and my VM and indeed the entire hypervisor ground to a halt (!). I don't think this was anything to do with the plugin thankfully, just a freak coincidence!

I ran another full sync, but those three movies were still listed as "missing" in the report. I thought this might be a cache issue, so I removed Plug-in Support\Caches\com.plexapp.plugins.trakttv and re-ran a sync, but to no avail. Looking over the log however, it looks like i'm grabbing an older version;

2016-08-04 22:54:14,162 - oem_client_provider_release.incremental (7f9e98d10700) : INFO (oem_client_provider_release.incremental:166) - [anidb -> imdb] Collection has been updated to v1.16.7

OpenEntityMap/oem-database-anidb-imdb@5e23b61 seems to have a version of v1.16.8 instead.

fuzeman commented 8 years ago

@KingJ Forgot to push the tags for the new updates yesterday. The updates should be published now, you can restart Plex Media Server to force an update check.

KingJ commented 8 years ago

@fuzeman Restarted, triggered a sync and logs say the mapping collection was updated. Those 3 movies are now mapping without issue :smile: .

iTorrek commented 8 years ago

New beta didn't help too guid="com.plexapp.agents.hama://tvdb2-81797/17/112?lang=en"

KingJ commented 8 years ago

@fuzeman Noticed that scrobbling stopped working today after whilst watching an episode backed by the TVDB agent rather than the Hama agent. In the channel menu, an exception is logged from yesterday, the full trace from the log is;

2016-08-06 20:34:02,817 - plugin.sync.main                 (7feb4effd700) :  WARNING (plugin.sync.main:163) - Exception raised in run(): HTTPSConnectionPool(host='api-v2launch.trakt.tv', port=443): Max retries exceeded with url: /sync/history (Caused by ReadTimeoutError("HTTPSConnectionPool(host='api-v2launch.trakt.tv', port=443): Read timed out. (read timeout=24)",))
Traceback (most recent call last):
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/plugin/sync/main.py", line 159, in run_wrapper
    self.run()
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/plugin/sync/main.py", line 246, in run
    'stop'
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/plugin/sync/main.py", line 263, in _trigger
    func()
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/elapsed/main.py", line 64, in inner
    return func(*args, **kwargs)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/plugin/sync/modes/full.py", line 46, in finish
    self.current.artifacts.send()
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/plugin/sync/core/task/artifacts.py", line 30, in send
    self.send_actions()
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/elapsed/main.py", line 64, in inner
    return func(*args, **kwargs)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/plugin/sync/core/task/artifacts.py", line 47, in send_actions
    self.send_action(data, action, **request)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/plugin/sync/core/task/artifacts.py", line 101, in send_action
    response = func(kwargs)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/trakt/interfaces/base/__init__.py", line 20, in wrap
    return func(*args, **kwargs)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/trakt/interfaces/sync/core/mixins.py", line 54, in add
    authenticated=kwargs.pop('authenticated', None)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/trakt/core/http.py", line 129, in post
    return self.request('POST', path, params, data, **kwargs)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/trakt/core/http.py", line 84, in request
    return self.send(prepared)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/trakt/core/http.py", line 102, in send
    response = self.session.send(request, timeout=timeout)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='api-v2launch.trakt.tv', port=443): Max retries exceeded with url: /sync/history (Caused by ReadTimeoutError("HTTPSConnectionPool(host='api-v2launch.trakt.tv', port=443): Read timed out. (read timeout=24)",))

There's also this exception raised as I started watching a show;

2016-08-07 19:21:16,228 - plugin.managers.client           (7feb4dffb700) :  INFO (plugin.managers.client:112) - Unable to find client with key 'REMOVED'
2016-08-07 19:21:24,145 - plugin.managers.action           (7feb4dffb700) :  WARNING (plugin.managers.action:75) - Unable to queue event 'scrobble/start' for <Session session_key: u'92:8', state: 'start'>: ConstraintError: UNIQUE constraint failed: action.queue.session_id, action.queue.event
Traceback (most recent call last):
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/plugin/managers/action.py", line 71, in queue
    queued_at=datetime.utcnow()
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/peewee.py", line 4088, in create
    inst.save(force_insert=True)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/peewee.py", line 4239, in save
    pk_from_cursor = self.insert(**field_dict).execute()
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/peewee.py", line 2945, in execute
    cursor = self._execute()
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/peewee.py", line 2455, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/playhouse/apsw_ext.py", line 106, in execute_sql
    raise
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/peewee.py", line 3009, in __exit__
    reraise(new_type, new_type(*exc_value.args), traceback)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/playhouse/apsw_ext.py", line 103, in execute_sql
    self._execute_sql(cursor, sql, params)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Trakttv.bundle/Contents/Libraries/Shared/playhouse/apsw_ext.py", line 95, in _execute_sql
    cursor.execute(sql, params or ())
  File "src/cursor.c", line 236, in resetcursor
IntegrityError: ConstraintError: UNIQUE constraint failed: action.queue.session_id, action.queue.event
fuzeman commented 8 years ago

@iTorrek The plugin doesn't support those identifiers yet, only com.plexapp.agents.hama://<anidb_id> identifiers are supported.

I'll have a look later this week, not sure how hard it would be to support these identifiers yet...

@KingJ Those warnings can be ignored, they usually only happen during high Trakt.tv API load.

Max retries exceeded with url: /sync/history

This error is fairly common, it just means the plugin couldn't connect to the Trakt.tv API. This usually just happens when the Trakt.tv API is overloaded.

IntegrityError: ConstraintError: UNIQUE constraint failed: action.queue.session_id, action.queue.event

This error means a scrobble action (start, stop, pause) has already been queued, probably should replace this with a more helpful message.

fuzeman commented 8 years ago

@iTorrek Support for Hama "tvdb2-" and "tvdb3-" identifiers is now available in the latest beta branch update. If there is any issues, please post the latest plugin log file.

fuzeman commented 8 years ago

Finally released in v1.1.0.6.

KingJ commented 8 years ago

Awesome! Many many thanks for working on this, it's great to see it released! :smile: