Closed almightiest closed 2 years ago
Hi, Thanks for your report. i have to dig into what xbmcnfotv://79089 refers to, i think possibly thetvdb entry? could you post example of one of your NFO files, one for movies and one for tv
I was able to fix nfo agents for plex, however for jellyfin, i would like to see more information as i have no clue how jellyfin represent nfo data into the db, could you go to one of your unsynced items and click edit metadata and tell me what kind of external IDs present? if it's natively supported it should list the external IDs into imdb field tvdb etc.
79089 is the id, and in this case it's for thetvdb for all tv series.
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<episodedetails>
<title>Offred</title>
<showtitle>The Handmaid's Tale</showtitle>
<rating>8.300000</rating>
<votes>5780</votes>
<epbookmark>0</epbookmark>
<year>2017</year>
<top250>0</top250>
<season>1</season>
<episode>1</episode>
<displayseason>-1</displayseason>
<displayepisode>-1</displayepisode>
<outline></outline>
<plot>Offred, one the few fertile women known as Handmaids in the oppressive Republic of Gilead, struggles to survive as a reproductive surrogate for a powerful Commander and his resentful wife. </plot>
<tagline></tagline>
<runtime>58</runtime>
<thumb>http://thetvdb.com/banners/episodes/321239/5883022.jpg</thumb>
<mpaa>TV-MA</mpaa>
<playcount>1</playcount>
<lastplayed>2018-03-20</lastplayed>
<file></file>
<path>smb://path/to/tv/The.Handmaids.Tale/</path>
<filenameandpath>smb://path/to/tv/The.Handmaids.Tale/The.Handmaids.Tale.S01E01.720p.BluRay.x264.mkv</filenameandpath>
<id>5883022</id>
<uniqueid type="tvdb" default="true">5883022</uniqueid>
<uniqueid type="imdb">tt5931646</uniqueid>
<premiered>2017-04-26</premiered>
<status></status>
<code></code>
<aired>2017-04-26</aired>
<studio>Hulu</studio>
<trailer></trailer>
<fileinfo>
<streamdetails>
<audio>
<channels>6</channels>
<codec>none</codec>
<language>eng</language>
<longlanguage>English</longlanguage>
</audio>
<video>
<aspect>1.778</aspect>
<codec>none</codec>
<durationinseconds>3443</durationinseconds>
<height>720</height>
<scantype>Progressive</scantype>
<width>1280</width>
</video>
</streamdetails>
</fileinfo>
<actor>
<name>Elisabeth Moss</name>
<role>June Osborne / Offred</role>
<thumb>http://thetvdb.com/banners/actors/422521.jpg</thumb>
</actor>
<actor>
<name>Max Minghella</name>
<role>Nick Blaine</role>
<thumb>http://thetvdb.com/banners/actors/404122.jpg</thumb>
</actor>
<actor>
<name>Joseph Fiennes</name>
<role>Commander Fred Waterford</role>
<thumb>http://thetvdb.com/banners/actors/404123.jpg</thumb>
</actor>
<actor>
<name>O-T Fagbenle</name>
<role>Luke Bankole</role>
<thumb>http://thetvdb.com/banners/actors/404124.jpg</thumb>
</actor>
<actor>
<name>Ann Dowd</name>
<role>Aunt Lydia</role>
<thumb>http://thetvdb.com/banners/actors/422101.jpg</thumb>
</actor>
<actor>
<name>Alexis Bledel</name>
<role>Emily / Ofglen / Ofsteven</role>
<thumb>http://thetvdb.com/banners/actors/422522.jpg</thumb>
</actor>
<actor>
<name>Yvonne Strahovski</name>
<role>Serena Joy Waterford</role>
<thumb>http://thetvdb.com/banners/actors/422523.jpg</thumb>
</actor>
<actor>
<name>Samira Wiley</name>
<role>Moira</role>
<thumb>http://thetvdb.com/banners/actors/422524.jpg</thumb>
</actor>
<actor>
<name>Madeline Brewer</name>
<role>Janine / Ofwarren / Ofdaniel</role>
<thumb>http://thetvdb.com/banners/actors/422529.jpg</thumb>
</actor>
<actor>
<name>Nina Kiri</name>
<role>Alma / Ofrobert</role>
<thumb>http://thetvdb.com/banners/actors/466359.jpg</thumb>
</actor>
<actor>
<name>Jenessa Grant</name>
<role>Dolores / Ofsamuel</role>
<thumb>http://thetvdb.com/banners/actors/466360.jpg</thumb>
</actor>
<actor>
<name>Amanda Brugel</name>
<role>Rita / Martha</role>
<thumb>http://thetvdb.com/banners/actors/481149.jpg</thumb>
</actor>
<actor>
<name>Clea DuVall</name>
<role>Sylvia</role>
<thumb>http://thetvdb.com/banners/actors/490545.jpg</thumb>
</actor>
<actor>
<name>Bradley Whitford</name>
<role>Commander Lawrence</role>
<thumb>http://thetvdb.com/banners/actors/545399.jpg</thumb>
</actor>
<resume>
<position>0.000000</position>
<total>0.000000</total>
</resume>
</episodedetails>
Thanks great, can you go and edit one episode meta data and see if the external ID present in jellyfin? for plex the fix is already out and the containers are building right now.
@pgjensen the new container builds are out, pull the new container and restart and try to import plex, it should just work now of course you have to set the -f
flag to import from beginning not just since last sync.
Thanks great, can you go and edit one episode meta data and see if the external ID present in jellyfin? for plex the fix is already out and the containers are building right now.
hmm even for the episodes that are adding successfully in jellyfin, there is no external id present in the metadata when i go into the GUI and Edit Metadata... there must be some other id internally it uses?
hmm even for the episodes that are adding successfully in jellyfin, there is no external id present in the metadata when i go into the GUI and Edit Metadata... there must be some other id internally it uses?
Internal ids are really not important, however they cannot be used to determine the status of the item vs your other source, because some items can be id 1 in plex and id 2 in jellyfin therefore it's impossible to tell, if you go to one of your items and click identify then try edit metadata do you see external IDs?
from what I've seen and read so far jellyfin seems to take the NFO as reference for external IDs to match items, it's completely weird not filling your external IDs sources
@almightiest the new container builds are out, pull the new container and restart and try to import plex, it should just work now of course you have to set the
-f
flag to import from beginning not just since last sync.
Plex is fixed for movies, but now for episodes it only adds a single episode per show and not all of the individual episodes.
This is the only entry I get for Handmaid's Tale
[2022-04-22T17:47:33+00:00] logger.DEBUG: [M: 13.65M] Adding episode [(via: plex), (series: The Handmaid's Tale), (year: 2017), (season: 1), (episode: 10), (title: Night), (date: 2017-06-14)]
Hmm, for movies i can see that they are unique ids probably, however for tv shows, could you for example go to your (The Handmaid's Tale) episodes .nfo files and see the uniqueid property in at least few episodes if they all match same id, then im afraid that means it will be impossible for this tool to parse the ids and link them correctly. we use the unique id of episodes/movies to do the linking between media sources.
Hmm, for movies i can see that they are unique ids probably, however for tv shows, could you for example go to your (The Handmaid's Tale) episodes .nfo files and see the uniqueid property in at least few episodes if they all match same id, then im afraid that means it will be impossible for this tool to parse the ids and link them correctly. we use the unique id of episodes/movies to do the linking between media sources.
The unique ids are all unique in the NFO files, no matter if it's a movie, tvshow, or episode. For instance, each uniqueid for a tvshow.nfo for all shows are unique for TheTVDb. Each uniqueid for an episode nfo is unique for TheTVDb pointing to the id of the episode itself, and not the tvshow id. Movies have uniqueid default set to imdb and are all unique.
For example, that Handmaid's Tale episode nfo points to id 5883022, and when you navigate to thetvdb it goes straight to the episode: https://thetvdb.com/series/the-handmaids-tale/episodes/5883022
That's good it hear, it means fix is possible however not from my side, as i can only deal with what is given to us, from looking at your earlier log
[2022-04-22T14:10:55+00:00] logger.NOTICE: [M: 6.72M] Ignoring plex - (TV Shows) - (episode:1) - [Samurai Champloo - (1x1)]. No valid GUIDs. [(0: {"id":"com.plexapp.agents.xbmcnfotv://79089/1/1?lang=xn"})]
[2022-04-22T14:10:55+00:00] logger.NOTICE: [M: 6.72M] Ignoring plex - (TV Shows) - (episode:1) - [Samurai Champloo - (1x2)]. No valid GUIDs. [(0: {"id":"com.plexapp.agents.xbmcnfotv://79089/1/2?lang=xn"})]
both of those episodes are reporting exactly same id 79089
which in this case reference the show title id not the episode it self, i believe the fix lies in agent to fix the unique id. usually ids are identified like this ://(id)
however for some reason the plex NFO agent seems to ignore episode id and choose to use human readable id instead of unique id.
Also could you link here the agent you are using for plex NFO parsing? i found one but seems to be abandoned ?
https://github.com/gboudreau/XBMCnfoTVImporter.bundle https://github.com/gboudreau/XBMCnfoMoviesImporter.bundle
both of those episodes are reporting exactly same id 79089 which in this case reference the show title id not the episode it self, i believe the fix lies in agent to fix the unique id. usually ids are identified like this ://(id) however for some reason the plex NFO agent seems to ignore episode id and choose to use human readable id instead of unique id.
Ahh I see... so their "unique id" is actually {tvshowid}/{seasonnum}/{episodenum} concatenated together, which of course would not translate between other servers like jellyfin/emby.
An alternative matching strategy could be to allow for matching to show id + season num + episode num as an alternative to purely unique ids. I believe another similar utility to this did that as a workaround to get these plex agents functioning. https://github.com/Taxel/PlexTraktSync/blob/main/plextraktsync/plex_api.py#L63
It's indeed possible to use the given format, to match, however the problem lies in the other media sources and how to reconcile the id back to them. the normal plex/jellyfin agents does not report tvshow id, as such you would have duplicate entries in your database. i think I'll have to sleep on it and try to figure out a way or suggest a fix for the agent importer.
i tried to come up with a way to fix this and it's not possible unless we re-write huge part of the tool.
I posted a feature request to the agent developer repo
https://github.com/gboudreau/XBMCnfoTVImporter.bundle/issues/123
unless this is implemented i dont see anyway for us to proceed in fixing this issue at the moment.
@pgjensen in the meantime until hopefully the maintainer fixes the problem could you mail me at ws-contact at arabcoders.net
your (The Handmaid's Tale) tvshow.nfo and the episodes nfo files as well. i want to see if i can provide patch.
Also please add this to your docker-compose.yaml
environment:
WS_DEBUG_IMPORT: 1
restart the container and run
docker exec -ti watchstate console state:import -f -vvr --servers-filter plex
this will generate log files in config/debug/ pick few of them and add them to the zip file.
After extensive testing, it's not possible currently to support this kind of GUIDs with the way we do syncing, as i mentioned in #83 i will probably do complete re-design in v2.x to support this use case.
Some updates to hopefully support this GUIDs in the future has been implemented in #89, #92, #93, #94, #95, #96
@pgjensen Hello, I have implemented basic support relative GUIDs. in 30min pull the new containers and try plex and jellyfin. Emby is untested for webhook as i no longer have premium license
I'm having issues with this and plex tv shows. It only imports 311 tv episodes when it should be in the thousands.
@almightiest
I'm having issues with this and plex tv shows. It only imports 311 tv episodes when it should be in the thousands.
Please open new issue and post some debug logs to see why the items aren't being imported, i suspect it's due to unsupported agent. if you post sample of your import debug logs specially the ones that contains Ignoring keyword it will help track down the issue.
Sorry it seems you have changed your username, thought it was someone different, anyhow, if you can pull the this container arabcoders/watchstate:dev-latest
and try it. i have many fixes that are pending until i finish migrating everything to the new database design.
It's not working but it may be the way i tried to switch to the dev container. It only matches on episode 1 of the first season it scans in a series. It won't match any more than 1 episode per series.
I docker-compose down
, removed the database, switched to the origin/dev branch, pulled, then did docker-compose up -d
and ran docker exec -ti watchstate console state:import -vvvrm -s plex -f > plex.log
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Adding plex - (TV Shows) - (episode:1) - [The Office - (2x1) - Merger]. As new Item.
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/2/2?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/2/2?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Ignoring plex - (TV Shows) - (episode:1) - [The Office - (2x2) - Appraisals]. State unchanged.
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/2/3?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/2/3?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Ignoring plex - (TV Shows) - (episode:1) - [The Office - (2x3) - Party]. State unchanged.
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/2/4?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/2/4?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Ignoring plex - (TV Shows) - (episode:1) - [The Office - (2x4) - Motivation]. State unchanged.
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/2/5?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/2/5?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Ignoring plex - (TV Shows) - (episode:1) - [The Office - (2x5) - Charity]. State unchanged.
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/2/6?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/2/6?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Ignoring plex - (TV Shows) - (episode:1) - [The Office - (2x6) - Interview]. State unchanged.
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/1/1?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/1/1?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.79M] Ignoring plex - (TV Shows) - (episode:1) - [The Office - (1x1) - Downsize]. State unchanged.
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/1/2?lang=xn)]
[2022-05-12T11:36:35+00:00] logger.DEBUG: [M: 12.78M] Parsing Legacy plex content agent. [(guid: com.plexapp.agents.xbmcnfotv://78107/1/2?lang=xn)]
Those messaged about parsing legacy agents has been removed a while back. which lead me to believe there is something wrong with the github automatic builds. If you run this command
docker exec -ti watchstate console state:import -vvvrf --deep-debug -s plex
Does it error out or completes. and could you inspect the files in /config/db/ for file named watchstate_v0.db if the file not found it means something either wrong with the auto container build or somehow you are not getting latest images.
Those messaged about parsing legacy agents has been removed a while back. which lead me to believe there is something wrong with the github automatic builds. If you run this command
docker exec -ti watchstate console state:import -vvvrf --deep-debug -s plex
Does it error out or completes. and could you inspect the files in /config/db/ for file named watchstate_v0.db if the file not found it means something either wrong with the auto container build or somehow you are not getting latest images.
I didn't save the edit on the docker-compose.yaml whoops. I watched it pull the dev-latest image this time.
I just re-ran it and it added a ton more episodes than before. I have noticed when I look at the database that it is adding some episodes more than once and some shows' episodes have multiple years listed, but don't know if that's going to cause any issues or not. I'll play around with it next week to see.
@almightiest Start with clean database, and run the same command I gave in the pervious post, please do attach the log and also give me the output of the duplicate show by running
docker exec -ti watchstate console db:list --type episode --title 'duplicate show name' --limit 1000 --output json
meta data such as title, year, via and extra sub keys are ignored when we compute changes that trigger an update.
The new changes #102 has been merged into master and should be on arabcoders/watchstate:latest
in 30min
@almightiest did you try the new builds if it fixes your problem?
arabcoders/watchstate:latest
I think so! Really appreciate the work. Gonna do some more testing, but everything imports at least now from what I can tell.
@almightiest
I think so! Really appreciate the work. Gonna do some more testing, but everything imports at least now from what I can tell.
That's great to hear, i'll be closing this issue for now. feel free to open another issue if you encounter any problems.
I'm having issues with nothing from Plex getting imported, and only a small fraction of Jellyfin episodes. Jellyfin movies seem fine.
In Plex, I use a local agent since I write my own NFO files for my owned media, and their agents don't support local NFO files out of the box. I suspect this may be causing issues if some column isn't being populated for records when imported that watchstate relies on?
In Jellyfin, I use local NFO files, but that's supported out of the box. And it's working for all Movies and a few TV shows.
Here's the initial import command I tried:
Then it scrolls through all of my Plex entries, but nothing at all gets imported. Example (which are confirmed watched under my user in Plex):
Then it scrolls through all of my Jellyfin entries, but only a fraction of the show/episodes imported. This one is definitely in a watched state via the jellyfin gui:
Then I marked a single episode as watched and re-ran the import without "-f" option, but it was not imported. Most shows/episodes seem broken. The ones listed as "updating episode" were not updated, and these are marked as updating every time i run import no matter what:
Jellyfin Movies seem fine, however. I updated a single movie as watched and re-ran without the "-f" option for jellyfin. It successfully imported the one I marked as watched, but it was odd that it updated 2 other episodes (and does the same ones every time i re-run):