Filebot Xattr series id support for AniDB, TVDB, TheMovieDb series

[ZeroQI]

Filebot support metadata in file system extended attributes, and has released a movies, series scanner and secondary agent here: https://github.com/filebot/plex-agents/blob/master/Scanners/Series/FileBot%20Xattr%20Series%20Scanner.py

One can instruct to save metadata in the OS file system metadata (examples attached)

• "database":"TheTVDB", "type":"TV Series","id":253463 => translate into [tvdb-id]
• "database":"AniDB", "type":"Anime", "id":578 => translate into [anidb-id
• "database":"TheMovieDB::TV", "type":"TV Series","id":42009 => translate into [tsdb-id]
• "database":"TheMovieDB::??????","type":"Movies", "id":????? => translate into [tmdb-id] {"@type":"Movie","year":2016,"imdbId":2513074,"tmdbId":293767,"language":"en","id":293767,"name":"Billy Lynn's Long Halftime Walk","aliasNames":[]}

xattr.zip

[top]

try:     import 'xattr.py' #from https://github.com/filebot/plex-agents, needs the scanner from FileBot installed
except:  FilebotID = {}
else:     FileBot       = {'TheTVDB': 'tvdb', 'AniDB': 'anidb', 'TheMovieDB::TV': 'tsdb', 'movie': 'tmdb'}

[ Line 647 before insertion above https://github.com/ZeroQI/Absolute-Series-Scanner/blob/master/Scanners/Series/Absolute%20Series%20Scanner.py]

  else:
    if Filebot:
      attr, db, sid = None
      for file in files:
        attr = xattr_metadata(file)
        if attr:
          db, sid = series_id(attr)  # db = attr_get(attr, 'seriesInfo', 'database')  # id = attr_get(attr, 'seriesInfo', 'id')  
          #if attr.get('imdbId'):  db, id = 'movie', movie_id(attr)
          if db in FileBot and sid:  # movies: mid = movie_id(attr), imdb_id = attr.get('imdbId'),tmdb_id = attr.get('tmdbId')
            source,id = FileBot[db], sid
            if source=='tvdb':  folder_show = id  #or source=='tmdb' in movie scanner
            else:               folder_show = "%s [%s-%s]" % (clean_string(reverse_path[0]), source, id)
            break

To do

• install FileBot scanner above to use 'xattr.py' in scanner Commons folder
• Include and test the code above
• add the necessary files to filebot, select the desired database (tmdb, anidb, tvdb) and filebot offers to automatically rename the files, but you don't have to , and thus FileBot add metadata for one selected database.
• Modify HAMA to support tvdb id as title (length/format test title for digit only series)

[ZeroQI]

Sven, are you using filebot ? If so, any way to test the code above ?

[sven-7]

I do use Filebot, but I'm trying to figure out what this would do to help?

[ZeroQI]

Filebot can save hidden (or not) metadata alongside the files The scanner could inherit the anidb or tvdb id automatically and pass it in the title to the agent that will not need updating....

I need a test to prove it inherits it to put the code in production but do not use filebot ...

[sven-7]

So the code gets inserted before this section? If so, I tried that and the scanner stopped working.

      else:
        Log.info('No forced id found in series folder name nor id file')
        source, id = "", ""
        folder_show = folder_show.replace(" - ", " ").split(" ", 2)[2] if folder_show.lower().startswith(("saison","season","series","Book","Livre")) and len(folder_show.split(" ", 2))==3 else clean_string(folder_show) # Dragon Ball/Saison 2 - Dragon Ball Z/Saison 8 => folder_show = "Dragon Ball Z"
    Log.info("".ljust(157, '-'))

[IIeTp]

Sorry for the long delay in answering. HAMA agent didn’t work for me, because the server was installed in a non-default directory and I couldn’t understand why nothing was working. I also tried to edit the scanner and nothing came of it, the scanner also stopped working. maybe we are doing something wrong? next time it’s better to give a link to the finished file to avoid errors during editing

[ZeroQI]

I modified the code and created a pull request in case it is not stable.

If somebody can test, it should get the ID from filebot files, and replace the title with thetvdb id if needed which thetvdb agent can use natively, and i wil need to add support for that in HAMA but that way agents don't need to be aware of Filebot Xattr to be supported :)

[ZeroQI]

Any feedback on the pull request?

[IIeTp]

Any feedback on the pull request?

I left a comment in the pull request topic. is it better to report on testing here?

[ZeroQI]

apologies, missed it, or more exactly i saw it but was on my phone then forgot about it...

The scanner code is not valid so it gives 
Apr 02, 2020 14:36:30.634 [3236] ERROR - No module in VideoFiles
Apr 02, 2020 14:36:30.634 [3236] ERROR - Error scanning directory .
Apr 02, 2020 14:36:30.635 [3236] ERROR - No module in Absolute Series Scanner
Apr 02, 2020 14:36:30.635 [3236] ERROR - We got an error scanning in W:\UNLIMITED\serials\webdl\lostfilm\1080p\Vikings 1 - LostFilm.TV [1080p]

if we replace

try:     import 'xattr.py' #from https://github.com/filebot/plex-agents, needs the scanner from FileBot installed
except:  FilebotID = {}
else:    FileBot   = {'TheTVDB': 'tvdb', 'AniDB': 'anidb', 'TheMovieDB::TV': 'tsdb', 'movie': 'tmdb'}

with FilebotID = {} and it works the issue is in the lines above, otherwise the rest of the changed lines

[IIeTp]

Absolute Series Scanner.py.txt

if I did everything right, then it didn’t work. same error

[ZeroQI]

Code was flawed

• Updated pull request code, bad variable name filebot/filebotID fixed, should work now
• need /Scanners/Common/xattr.py present but should not crash if not there

[IIeTp]

xattr.py.txt https://github.com/filebot/plex-agents/blob/master/Plug-ins/Xattr.bundle/Contents/Code/xattr.py answered in pull request branch. same error

[ZeroQI]

Updated scanner code in pull request

• Doesn't crash for me, was a space tabulation issue...
• Added xattr scanner and agents, doesn't crash neither
• pending forced id test through xattr

[ZeroQI]

Just fixed a typo Since it doesn't crash merged the pull request

• ASS takes the forced id first, and if not present the Xattr
• Needs Xattr scanner installed, especially /Scanners/common/xattr.py [https://github.com/filebot/plex-agents installed]
• Just fixed a typo, should work now

[IIeTp]

I ran the test again. the text file was called /test/Blindspot.S02E01.In.Night.So.Ransomed.Rogue.mkv but the information in .xattr was from https://thetvdb.com/series/billions s01e01.

The result was blindspot s02e01. scanner ignored .xattr

[ZeroQI]

I would need a copy of .xattr folder and filelist so i can reproduce on my side to be sure to fix it in one go

[IIeTp]

sure Lucifer.zip w:\UNLIMITED\serials\webdl\lostfilm\test\Lucifer\Lucifer.S01E01.Pilot.mkv

[ZeroQI]

Hi, thanks to your files, i noticed the errors in the code since i could reproduce Wasn't far off

• imported xattr in lieu of filebot
• attr, db, sid = None, "", "" was missing 2 vars initialization
• 'db, sid = filebot.seriesid(attr)' needing a split with '.split('')' to get two vars
• needed to prepend filebot to function calls: filebot.xattr_metadata, filebot.series_id

apart from that the code was good so the theory was sound... You however need a change in the folder for Plex to re-run the scanner as it cache results (even if it crashes which is stupid)...

In my test, the series hadthe tvdbid as title so that should make standard tvdb agent work seamlessly, need to add tvdbid as title support in Hama now...

Please test and report

[IIeTp]

test.zip w:\UNLIMITED\serials\webdl\lostfilm\test\Happy!.S01E06.The.Scrapyard.of.Childish.Things.mkv

same story. still not using information from .xattr

[ZeroQI]

Here are my tests with your files:

.\Logs\ASS Scanner Logs\Lucifer.scanner.log

=============================================================================================================================================================
Library: 'no valid X-Plex-Token.id', root: '/volume1/downloads/complete/test library series', path: 'Lucifer', files: '1', dirs: '0'
Plex scan start: 2020-04-18 00:00:31,066377
=============================================================================================================================================================
FileBot Xattr found, source: tvdb, id: 279536, attr: {u'episode': 1, u'seriesInfo': {u'status': u'Continuing', u'startDate': {u'month': 1, u'day': 17, u'year': 2016}, u'genres': [u'Drama'], u'rating': 8.6999999999999993, u'name': u'Billions', u'language': u'en', u'database': u'TheTVDB', u'certification': u'TV-MA', u'aliasNames': [u'Billions (2016)', u'Miliardy', u'\u05de\u05d9\u05dc\u05d9\u05d0\u05e8\u05d3\u05d9\u05dd', u'Milli\xe1rdok nyom\xe1ban', u'\ube4c\ub9ac\uc5b8\uc2a4', u'\u041c\u0438\u043b\u043b\u0438\u0430\u0440\u0434\u044b', u'\u4ebf\u4e07'], u'order': u'DVD', u'ratingCount': 979, u'runtime': 60, u'type': u'TV Series', u'id': 279536, u'network': u'Showtime'}, u'title': u'Pilot', u'season': 1, u'airdate': {u'month': 1, u'day': 17, u'year': 2016}, u'id': 4825218, u'seriesName': u'Billions', u'@type': u'Episode', u'absolute': 1}
-------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------
misc_count: {'Lucifer': 1, 'S01E01': 1, 'Pilot': 1}
misc_words: []
-------------------------------------------------------------------------------------------------------------------------------------------------------------
"279536" s01e001                         "SERIES_RX-1" "Pilot" "Lucifer.S01E01.Pilot.mkv"
=============================================================================================================================================================

.\Logs\ASS Scanner Logs\Test.scanner.log

=============================================================================================================================================================
Library: 'no valid X-Plex-Token.id', root: '/volume1/downloads/complete/test library series', path: 'Test', files: '1', dirs: '0'
Plex scan start: 2020-04-17 23:57:49,135529
=============================================================================================================================================================
FileBot Xattr found, source: tvdb, id: 292124, attr: {u'episode': 1, u'seriesInfo': {u'status': u'Ended', u'startDate': {u'month': 6, u'day': 14, u'year': 2015}, u'genres': [u'Drama', u'Science Fiction'], u'rating': 8.5999999999999996, u'name': u'Humans', u'language': u'en', u'database': u'TheTVDB', u'certification': u'TV-14', u'aliasNames': [], u'order': u'DVD', u'ratingCount': 1530, u'runtime': 45, u'type': u'TV Series', u'id': 292124, u'network': u'Channel 4'}, u'title': u'Episode 1', u'season': 1, u'airdate': {u'month': 6, u'day': 14, u'year': 2015}, u'id': 5129608, u'seriesName': u'Humans', u'@type': u'Episode', u'absolute': 1}
-------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------
misc_count: {'Scrapyard': 1, 'Happy!': 1, 'Childish': 1, 'S01E06': 1, 'of': 1, 'The': 1, 'Things': 1}
misc_words: []
-------------------------------------------------------------------------------------------------------------------------------------------------------------
"292124" s01e006                         "SERIES_RX-1" "The.Scrapyard.of.Childish.Things" "Happy!.S01E06.The.Scrapyard.of.Childish.Things.mkv"
=============================================================================================================================================================

The output seem good to me, the series name is changed by the tvdbid Hama will not handle well for now but thetvdb will

[IIeTp]

I checked with TVDB + ASS. The result is the same. And this is logical, the agent works very badly with id. I have tested it a lot before.

p.s. the default scanner TVDB is very poorly matched by id. If you put id instead of the name (only numbers), the scanner will not get any information in automatic mode in 70-85% of cases. If you try to correct the comparison, it will be seen that id "295685" has only 81 points, which will not allow him to match in automatic mode (we need 90+).

If you add the id "Lucifer 295685" to the name of the series, then the search does not find anything.

[ZeroQI]

I have amended Hama so it transform "295685" into "xxx [tvdb-295685]" to force the id... Please check the scanner logs to see if the title was replaced by the tvdbid With external Xattr it works for me, and Hama was amended but series with only numbers (24) will have issues i Hama but scanner ok

Edit: I have re-created the two series with real mkv file episode, and a test series with no meta .xattr folder present...

• normal folder ok
• other folders with .xattr folder inside merge in same series with "" title despite being added with title... Even if removing one .xattr folder and splitting the series, still no title

Plex Media Scanner.log

Apr 18, 2020 16:19:45.973 [0x7f00a684f740] DEBUG - Updating directory '' (ID 23449) to time 2020-04-18 16:18:56.
Apr 18, 2020 16:19:45.974 [0x7f00a684f740] DEBUG - Updating directory 'Lucifer' (ID 23450) to time 2020-04-18 16:16:31.
Apr 18, 2020 16:19:45.991 [0x7f00a684f740] DEBUG - Updating directory 'The 100' (ID 23451) to time 2020-04-18 16:16:15.
Apr 18, 2020 16:19:46.007 [0x7f00a684f740] DEBUG - Updating directory 'Title xxx' (ID 23452) to time 2020-04-18 16:16:45.
[...]
Apr 18, 2020 16:19:46.010 [0x7f00a684f740] DEBUG - The show '' needs matching, no refresh yet.
Apr 18, 2020 16:19:46.011 [0x7f00a684f740] DEBUG - The show 'The 100' needs matching, no refresh yet.

Note sure how to fix. seem like the same subfolder name confuses the backend

The .xattr folder presence make the serie be merged Tried stacking results with " Stack.Scan(path, files, media, dirs)" and "import Stack", no change Not sure how to solve, Plex seem buggy there

[ZeroQI]

@IIeTp support for TheTVDB ID as title was addedin april 2020 Any feedback since?

[IIeTp]

Sorry, but I already uninstalled Plex, so I can't add anything extra.

[ZeroQI]

Thanks for letting me know My tests were conclusive and thetvdbid supported as title so should work, and code included in master Updated readme, closing this ticket