Gintama Auto-Update Not Working

wilyub commented 11 months ago

I'm currently rewatching Gintama and am on Season 1 Episode 72 (S2 Episode 23 according to TVDB naming scheme). When I finished the episode, the plugin didn't update the episode on anilist. Here is the log information:

[2023-10-10 16:05:55.570 -04:00] [INF] [27] Emby.Server.Implementations.Session.SessionManager: Playback stopped reported by app "Jellyfin Web" "10.8.11" playing "A Dog's Paws Smell Fragrant / Drive with a \"Might\" Attitude". Stopped at "1469694" ms [2023-10-10 16:05:55.576 -04:00] [WRN] [27] Jellyfin.Server.Middleware.ResponseTimeMiddleware: Slow HTTP Response from "http://localhost:8096/Sessions/Playing/Stopped" to "127.0.0.1" in 0:00:01.5741719 with Status Code 204 [2023-10-10 16:05:55.589 -04:00] [INF] [29] jellyfin_ani_sync.UpdateProviderStatus: (Anidb) Anime already has AniDb ID; no need to look it up [2023-10-10 16:05:55.590 -04:00] [INF] [29] jellyfin_ani_sync.UpdateProviderStatus: (Anidb) Anime Gintama found in anime XML file [2023-10-10 16:05:55.591 -04:00] [INF] [29] jellyfin_ani_sync.UpdateProviderStatus: Retrieving provider IDs from offline database... [2023-10-10 16:05:55.667 -04:00] [INF] [36] jellyfin_ani_sync.UpdateProviderStatus: Retrieved provider IDs [2023-10-10 16:05:55.667 -04:00] [INF] [36] jellyfin_ani_sync.UpdateProviderStatus: Using provider AniList... [2023-10-10 16:05:55.808 -04:00] [INF] [36] jellyfin_ani_sync.UpdateProviderStatus: User does not have plan to watch only ticked [2023-10-10 16:05:55.809 -04:00] [INF] [36] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (Gintama) found, but provider reports episode already watched. Skipping [2023-10-10 16:05:55.809 -04:00] [INF] [36] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (Gintama) found on Rewatching list [2023-10-10 16:05:55.810 -04:00] [INF] [36] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Provider reports episode already watched; not updating [2023-10-10 16:05:55.810 -04:00] [ERR] [36] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Could not update anime status

My anilist only has Gintama Rewatching episode 71 finished, so I believe the plugin might be trying to set episode 23 as watched? I saw in the anime master list that Gintama Episodes for Seasons 1-4 (TVDB) do have a mapping to AniDB Season 1 (201 episodes). Is there some way to use that information so that it can still map an episode S02E23 to Gintama Season 1 Episode 72? Thanks!

vosmiic commented 11 months ago

Unfortunately this is probably another case of different providers formatting shows in different seasons/episodes. But I might be able to help, what season and episode format do you have set in Jellyfin? Is it season 1 episode 72 or season 2 episode 23?

wilyub commented 11 months ago

I have it set as Season 2 Episode 23 (that's what the jellyfin UI shows). My file title is "Gintama S02E23" with a folder structure as follows: "Gintama (2006)/Season 2/Gintama S02E23"

wilyub commented 11 months ago

I've been doing a little testing, and I believe the issue comes down to using AniDB vs. TVDB episode #. I can't say for sure cause I haven't seen it in the code yet, but if the logic you use to determine episode # extracts the TVDB episode #, that would explain why I'm running into an issue with shows like Gintama. Could you point me to the part of your code where you determine the episode number, and maybe I could write something up for testing purposes?

I think in general anime entries on Anilist are more likely to follow AniDB episode #s rather than TVDB numbers. For instance, Gintama has the first season as 201 episodes on Anilist, but 4 seasons on TVDB. Bungou Stray Dogs has 5 seasons on anilist, but 3 seasons on TVDB. Therefore, if we have a way of connecting AniDB Seasons w/ Anilist Seasons I think it would make more sense to have the episode # come from AniDB (rather than TVDB).

vosmiic commented 11 months ago

Here is the area we get the AniDB ID and we calculate the episode by using an offset (if the show is in the XML anime list): https://github.com/vosmiic/jellyfin-ani-sync/blob/f4c6fa981b42b8aa94c9bfeeb48ddc1d7a9a1ba2/Helpers/AnimeListHelpers.cs#L24-L85 Hopefully that helps. Its difficult to make the plugin work with all of the providers we support since they seem to all use a slightly different way of numbering episodes and seasons (as you have found out). Originally the plugin was designed for only MyAnimeList since its the service I use, so the other service's compatibility is not as strong as MAL's. That being said its usually not an issue for shows that have generic season and episode numbering.

wilyub commented 11 months ago

Thanks for your help. I traced the code to this line in the "UpdateProviderStatus.cs" file.

https://github.com/vosmiic/jellyfin-ani-sync/blob/f4c6fa981b42b8aa94c9bfeeb48ddc1d7a9a1ba2/UpdateProviderStatus.cs#L158C64-L158C64

This conditional checks for an episode offset value in the aniDbId variable. Does it check for an offset in the "anime" section of the anime-list xml or in the "mapping-list" section of the xml? For instance:

This line for Gintama is the "anime" section and has no offset: https://github.com/Anime-Lists/anime-lists/blob/master/anime-list.xml#L10050

This line for Gintama is the "mapping" section and has different offsets for each season: https://github.com/Anime-Lists/anime-lists/blob/master/anime-list.xml#L10055

If it's only looking for an offset in "anime", that could explain why the Jellyfin log tells me "provider reports episode already watched. Skipping". Since the episode would look like S02E23, it would try to mark Episode 23 as watched when I've already watched up to episode 71. I tried looking into this myself, but I can't figure out how to get the debugger working (I'm mostly a python guy xD).

Anyway, if this is the case, would it be possible to add a flag to check for this behavior? It seems like a lot of anime with the absolute order flag ("defaulttvdbseason="a") have this same problem (mapping one anidb season to multiple tvdb seasons).

wilyub commented 11 months ago

Ok I've done a bit more testing, some interesting results. I renamed the gintama episodes for season 1 (episodes 1 - 201) to be absolute order. So S01E72 is the episode I played. Using this naming scheme the episodes all show up under Season 1 in Jellyfin, and the plugin is able to successfully update Episode 72 on Anilist.

I then tried to watch Gintama' (2011) Episode 29. This is named "Gintama S05E29" in the "Season 5" folder. However, this did not update the Anilist entry. I looked through the logs and found this.

[2023-10-12 19:33:26.683 -04:00] [INF] [76] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (Gintama) found, but provider reports episode already watched. Skipping [2023-10-12 19:33:26.683 -04:00] [INF] [76] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (Gintama) found on Rewatching list [2023-10-12 19:33:26.684 -04:00] [INF] [76] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Provider reports episode already watched; not updating [2023-10-12 19:33:26.684 -04:00] [ERR] [76] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Could not update anime status

This leads me to believe that the plugin is only looking at the AniDB ID/TVDB ID for the original Gintama Season without looking at the season. For instance, in the XML file we see that the TVDB ID for Gintama and Gintama' (2011) are the same (TVDB has them as Seasons 1-4 and Season 5 respectively), however because the seasons are different we should be able to distinguish them to two separate AniDB IDs. This behavior should work the same whether it's MAL or Anilist because the entries have the same episode counts (Gintama == 201 episodes, Gintama'(2011) == 51 episodes).

I think the logic I would use for determining the exact Anilist/MAL entry would be as follows:

Find TVDB ID & Season number (Should be in the Jellyfin Metadata).
Go to anime-list.xml and find all entries that have a matching TVDB ID.
- If the list is size 1, then use the old logic and we're done.
- If the list is size > 1, then iterate through the entries by looking at the "defaulttvdbseason" category in the "anime" node.
- - If the season number matches the entry, use that AniDB ID and then follow the old logic. We're done.
- - If the season number is "a" (absolute ordering), then go into the "mapping-list" node and iterate through the seasons there until you find a matching season. (Example: Fairy Tail (2014)). Once you confirm the season # is in there, use the AniDB from the parent "anime" node and follow old logic. We're done.

For "old logic", I'm just assuming we match up the AniDB ID to the MAL/Anilist ID by using the arm server repo on the external tools section.

Unfortunately (as I mentioned earlier), I have little experience with C# and the Jellyfin backend, so I don't really know how to code this all myself. But if you happen to find some free time, please give this a look and see if it would make sense to implement. Thanks for the great plugin!

vosmiic commented 11 months ago

Sure, will investigate this. Planning on starting up development again of the plugin now as there are quite a few things to look at, and will make this one of them. Thanks for the detailed analysis, will make implementing any changes much faster. I'll keep you updated on the progress 👍

wilyub commented 11 months ago

Thank you very much! Let me know if you need any help testing stuff. I'm happy to mess around with filenames/metadata to try and get to the bottom of this :D

KaXaSA commented 11 months ago

I guess I'm having the same issue, but with One Piece, other stuff has been working flawlessly. I'm not sure if this useful information, but:

I use Sonarr to deal with metadata, folder and file management. Sonarr uses TVDB as their metadata source. My folders/file structure look like this:

One Piece (1999)
├── Season 21
│   ├── One Piece (1999) - S21E185 - 1076 - Episode Title.mkv

1076 = episode absolute number

Some of the episode metadata copied directly from Jellyfin:

Season number: 21
Episode number: 185
AniDB Id: 269850
TheTVDB Episode Id: 10024783

In Jellyfin this looks absolutely perfect, I don't see anything wrong. The anime, season and episode are all correct.

Test:

Open Anilist, change the Episode Progress to 1
Open Jellyfin and mark as watched: Season 21 > Episode 185 (you can test using any episode from season 2+)

Log:

[INF] [75] jellyfin_ani_sync.UpdateProviderStatus: (Anidb) Anime already has AniDb ID; no need to look it up
[INF] [75] jellyfin_ani_sync.UpdateProviderStatus: (Anidb) Anime One Piece found in anime XML file
[INF] [75] jellyfin_ani_sync.UpdateProviderStatus: Retrieving provider IDs from offline database...
[INF] [80] jellyfin_ani_sync.UpdateProviderStatus: Retrieved provider IDs
[INF] [80] jellyfin_ani_sync.UpdateProviderStatus: Using provider AniList...
[INF] [80] jellyfin_ani_sync.UpdateProviderStatus: User does not have plan to watch only ticked
[INF] [80] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (ONE PIECE) found on Plan_to_watch list
[INF] [80] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Updated series (ONE PIECE) progress to 185

So it changed the Episode Progress in Anilist to 185 instead of changing it to 1076

Basically, the way it works now, I would have to change the metadata of all the episodes to season 1 and replace the episode number with the absolute episode number Using my example (S21E185 - 1076) I would need to change the episode metadata:

From
Season number: 21
Episode number: 185

To
Season number: 1
Episode number: 1076

Then it should work with ani sync.

SolarWolf-Code commented 9 months ago

@KaXaSA do you mind sending your Sonarr Anime Episode Format that you got it working with?

KaXaSA commented 9 months ago

@KaXaSA do you mind sending your Sonarr Anime Episode Format that you got it working with?

Sadly, I actually didn't get it working at all, you would need to manually change the metadata or create a script to do a search & replace on the metadata files to change the season of all the episodes to 1.

I don't think you can change the episode format nor any other option in Sonarr to easily fix this.

SolarWolf-Code commented 9 months ago

@KaXaSA do you mind sending your Sonarr Anime Episode Format that you got it working with?

Sadly, I actually didn't get it working at all, you would need to manually change the metadata or create a script to do a search & replace on the metadata files to change the season of all the episodes to 1.

I don't think you can change the episode format nor any other option in Sonarr to easily fix this.

Okay some good news. I think I've got something working :))

Here are the steps:

Set Sonarr Anime Episode Format

We can actually force Sonarr to name the episode S01 every time. You'll notice instead of using something more traditional such as S{season:00} we use S01. You can simply paste this into the Anime Episode Format located at Settings > Media Management.

{Series TitleYear} - S01E{absolute:000} - {Episode CleanTitle} [{Custom Formats }{Quality Full}]{[MediaInfo VideoDynamicRangeType]}[{MediaInfo VideoBitDepth}bit]{[MediaInfo VideoCodec]}[{Mediainfo AudioCodec} { Mediainfo AudioChannels}]{MediaInfo AudioLanguages}{-Release Group}

This results in a naming scheme like this:

Single Episode: The Series Title! (2010) - S01E001 - Episode Title 1 [HDTV-720p v2][HLG][10bit][x264][DTS 5.1][JA]-RlsGrp
Multi Episode: The Series Title! (2010) - S01E001-003 - Episode Title [HDTV-720p v2][HLG][10bit][x264][DTS 5.1][JA]-RlsGrp

There are a few other things that I have setup to make this work. Firstly, we need to make sure and select Anime as the series type:
That will force the format above.

I also like having season folders. Series Folder Format: {Series TitleYear} [tvdbid-{TvdbId}]. Tvdbid is not required but having it helps Jellyfin a ton in my experience

Season Folder Format: Season {season:00} The directory structure will look like this:

Gintama (2006) [tvdbid-79895]/
├── Season 01
│   ├── Gintama (2006) - S01E001-002 - You Guys!! Do You Even Have Gintama!! [HDTV-1080p][10bit][x265][Opus 2.0][JA]-Judas.mkv
│   ├── Gintama (2006) - S01E003 - Nobody with Naturally Wavy Hair Can Be That Bad [HDTV-1080p][10bit][x265][Opus 2.0][JA]-Judas.mkv
│   └── Gintama (2006) - S01E004 - Watch Out! Weekly Shonen JUMP Sometimes Comes Out on Saturdays! [HDTV-1080p][10bit][x265][Opus 2.0][JA]-Judas.mkv
└── Season 02
    ├── Gintama (2006) - S01E050 - Pending Means Pending Its Not Final [HDTV-1080p][10bit][x265][Opus 2.0][JA]-Judas.mkv
    ├── Gintama (2006) - S01E051 - Milk Should Be Served at Body Temperature [HDTV-1080p][10bit][x265][Opus 2.0][JA]-Judas.mkv
    └── Gintama (2006) - S01E052 - If You Want to See Someone Make an Appo First [HDTV-1080p][10bit][x265][Opus 2.0][JA]-Judas.mkv

How it appears in Jellyfin

The season folders properly show up:
And the metadata is properly filled in:
Season 2 title will appear as absolute episode numbers:

Testing it with AniList

Firstly, make sure that you have this plugin "safely" set up. To do so, ensure that you have the following option ticked:

After we watch episode 23 of Season 2, absolute episode 72 (A Dog's Paws Smell Fragrant / Drive with a "Might" Attitude) we can see the plugin properly set the episode as watched:

[!NOTE] I've tested this with Gintama first two seasons and Jujutsu Kaisen, so I'm not sure it works with every anime quite yet!

KaXaSA commented 9 months ago

@SolarWolf-Code Try to test that with a more recent anime, to see if it will display the proper episode information for seasons 2+

SolarWolf-Code commented 9 months ago

@KaXaSA I've tried with Jujutsu Kaisen as well but I'll give another few a try :))

KaXaSA commented 9 months ago

@SolarWolf-Code I tested with some stuff I have here, it seems to works fine on jellyfin, but One Piece specifically still doesn't work with the ani-sync plugin:

...
[22:33:03] [INF] [26] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (ONE PIECE) found on Plan_to_watch list
[22:33:03] [INF] [26] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Provider reports episode already watched; not updating
[22:33:03] [ERR] [26] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Could not update anime status

SolarWolf-Code commented 9 months ago

Well it seemed to have worked for Gintama but it doesn't seem to be working for some other anime. Despite the episodes being in separate season folders, the metadata seems to be filling the season based on the name. Not only this but, sometimes AniSync sends a request past the amount of episodes:

The way AniList splits the anime is what can cause problems. For long running shows without breaks they will have a single entry for it. Although for a show like AoT that is not the case

A possible solution to this is use the different formats. Such as Standard for your standard shows, Anime for split up shows, and Daily as long running shows. Really not sure this is a good method though...

SolarWolf-Code commented 9 months ago

@SolarWolf-Code I tested with some stuff I have here, it seems to works fine on jellyfin, but One Piece specifically still doesn't work with the ani-sync plugin:
...
[22:33:03] [INF] [26] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (ONE PIECE) found on Plan_to_watch list
[22:33:03] [INF] [26] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Provider reports episode already watched; not updating
[22:33:03] [ERR] [26] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Could not update anime status

Do you have Automatically set completed shows as rewatching? ticked? I'm not 100% sure but the logs suggest its because you have it already watched?

Also could you provide the episode you are testing with? I can give it a go on my end as well

KaXaSA commented 9 months ago

Do you have Automatically set completed shows as rewatching? ticked? I'm not 100% sure but the logs suggest its because you have it already watched?

Nah, that's not it, I don't even have that episode marked as watched in Anilist, when I tested this the last time, I changed the episode progress to 1 in Anilist, then in Jellyfin I marked S21E185 (absolute: 1076) as watched, and the ani-sync plugin updated the anime progress to episode 185 (instead of 1076).

So that explains the Provider reports episode already watched; not updating it's ignoring the season and using the episode number as if it were the absolute episode number

A possible solution to this is use the different formats. Such as Standard for your standard shows, Anime for split up shows, and Daily as long running shows. Really not sure this is a good method though...

I tried to use Standard like that, but it turns out that Standard is only good for series, it doesn't work well with anime. Maybe it could work with Daily (it just doesn't have the {absolute...} format), but I suspect this wouldn't fix the issue with One Piece because the plugin will probably use the information available in the metadata file instead of using the file name, and in the metadata it still uses the TVDB season format.

Jellyfin does have a AniDB metadata plugin I wonder if that could help, I recall that I tested this in the past and had some issue that made me disable it and just use Sonarr's metadata stuff instead.

vosmiic commented 9 months ago

Now that I am looking at this again, I have had a look at the code I am found some interesting notes. Probably didn't think of these earlier as I haven't worked in this specific area of the plugin for some time.

I think the logic I would use for determining the exact Anilist/MAL entry would be as follows:

* Find TVDB ID & Season number (Should be in the Jellyfin Metadata).

* Go to anime-list.xml and find all entries that have a matching TVDB ID.

* * If the list is size 1, then use the old logic and we're done.

* * If the list is size > 1, then iterate through the entries by looking at the "defaulttvdbseason" category in the "anime" node.

* * * If the season number matches the entry, use that AniDB ID and then follow the old logic. We're done.

* * * If the season number is "a" (absolute ordering), then go into the "mapping-list" node and iterate through the seasons there until you find a matching season. (Example: Fairy Tail (2014)). Once you confirm the season # is in there, use the AniDB from the parent "anime" node and follow old logic. We're done.

It is actually already doing this (for TVDB IDs), in a slightly different way. On these lines (I understand you have limited exposure to C#, so I have attempted to explain how it works below): https://github.com/vosmiic/jellyfin-ani-sync/blob/55b5dc8dc189338c9f7b4cab37874e25fbcae1ce/Helpers/AnimeListHelpers.cs#L73-L81 We are looking up the anime in the XML, iterating through the results and finding which AniDB entry the current episode belongs to. So if you were watching episode 200 of Fairy Tail, it would assume you are watching "Fairy Tail (2014)" and send back 9980 as the AniDB ID, which is correct. Since we don't use the TVBD ID with the API database, we can disregard it. I think the root of your original problem is that you probably have a mixture of AniDB and TVDB metadata in your library entries (not a problem on your end, can't blame you for having too much metadata and the plugin should be able to deal with it). So I believe that your library layout (seasons and episodes) is using the TVDB format, while each entry has AniDB metadata.

This leads me to believe that the plugin is only looking at the AniDB ID/TVDB ID for the original Gintama Season without looking at the season

Not quite, but you aren't exactly incorrect. We do use the season number but it doesn't have much importance when looking up the AniDB entry in the XML, since the AniDB entry shouldn't depend on the season. Each AniDB entry is its own season (or at least should be), hence why the plugin prefers to use it over TVDB (which as you know likes to throw everything into a single TVDB entry). So the plugin gets confused; it doesn't care you are watching episode 200, it sees you have an AniDB ID in your metadata so it can simply look up the ID and return the only result (also if there is more than one entry in the XML list with the same AniDB ID the plugin falls back to other metadata providers if they exist). So @KaXaSA you are pretty much spot on with

So that explains the Provider reports episode already watched; not updating it's ignoring the season and using the episode number as if it were the absolute episode number

because it shouldn't need to look up seasons.

This needs changing, for these occasions, as you have outlined. Perhaps looking up the entry via the AniDB, checking if there are other entries that have the same TVDB ID. Then compare the offsets of the entries and get the AniDB of the correct entry. This sounds very similar to your original suggested logic changes @wilyub so apologies if I am basically reiterating it, but just wanted to voice my findings. That being said, I would suggest (for now) perhaps removing either the TVDB or AniDB metadata providers (and forcing a refresh (and select "Replace all metadata" for the refresh mode) of Gintama so the seasons/episodes get shuffled around in Jellyfin) as they obviously don't work well together and see if that fixes things. I would suggest using the AniDB metadata provider if you can, it generally works a bit better with the plugin. If you do decide to remove one of these metadata providers and refresh your metadata, please do let us know how you get on. If it fixes your issues its probably a good idea to let others know that using these two metadata providers at the same time causes issues. The wiki page here needs updating to reflect this, it hasn't changed since we overhauled how we detect anime.

Also thanks guys for digging quite deep into this to work out where the plugin needs improvements, if you have found anything that I haven't or believe any of what I have said above is incorrect please do correct me/let me know, like I said this is some old code so I may have missed something.

wilyub commented 9 months ago

Thanks for taking the time to look at this issue, I really appreciate it :)

I did a quick test on Gintama by only having TVDB metadata and unfortunately it still didn't work. This is the logs.

[2023-11-27 21:36:57.563 -05:00] [INF] [14] jellyfin_ani_sync.UpdateProviderStatus: (Tvdb) Anime reference found in anime list XML [2023-11-27 21:36:57.564 -05:00] [INF] [14] jellyfin_ani_sync.UpdateProviderStatus: (Tvdb) Anime Gintama found in anime XML file (using Tvdb ID) [2023-11-27 21:36:57.564 -05:00] [INF] [14] jellyfin_ani_sync.UpdateProviderStatus: Retrieving provider IDs from offline database... [2023-11-27 21:36:57.689 -05:00] [INF] [43] jellyfin_ani_sync.UpdateProviderStatus: Retrieved provider IDs [2023-11-27 21:36:57.689 -05:00] [INF] [43] jellyfin_ani_sync.UpdateProviderStatus: Using provider AniList... [2023-11-27 21:36:57.852 -05:00] [INF] [43] jellyfin_ani_sync.UpdateProviderStatus: User does not have plan to watch only ticked [2023-11-27 21:36:57.852 -05:00] [INF] [43] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (Gintama) found, but provider reports episode already watched. Skipping [2023-11-27 21:36:57.853 -05:00] [INF] [43] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (Gintama) found on Rewatching list [2023-11-27 21:36:57.853 -05:00] [INF] [43] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Provider reports episode already watched; not updating [2023-11-27 21:36:57.854 -05:00] [ERR] [43] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Could not update anime status

I would test out AniDB only, but it would require me renaming all my files so I'll just hold off for now and wait for the updated anime selection logic.

KaXaSA commented 9 months ago

I did a quick test on Gintama by only having TVDB metadata and unfortunately it still didn't work.

Yeah, same here, I tested it with One Piece (S21E187):

[00:32:06] [INF] [15] jellyfin_ani_sync.UpdateProviderStatus: (Tvdb) Anime reference found in anime list XML
[00:32:06] [INF] [15] jellyfin_ani_sync.UpdateProviderStatus: Retrieving provider IDs from offline database...
[00:32:06] [INF] [42] jellyfin_ani_sync.UpdateProviderStatus: Retrieved provider IDs
[00:32:06] [INF] [42] jellyfin_ani_sync.UpdateProviderStatus: Using provider AniList...
[00:32:07] [INF] [42] jellyfin_ani_sync.UpdateProviderStatus: User does not have plan to watch only ticked
[00:32:07] [INF] [42] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (ONE PIECE) found on Plan_to_watch list
[00:32:07] [INF] [42] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Updated series (ONE PIECE) progress to 187

S21E187 metadata:

<episodedetails>
  <title>redacted - spoilers</title>
  <season>21</season>
  <episode>187</episode>
  <aired>2023-10-01</aired>
  <plot>redacted - spoilers</plot>
  <uniqueid type="tvdb" default="true">10024785</uniqueid>
  <uniqueid type="sonarr">1851</uniqueid>
  <thumb>https://artworks.thetvdb.com/banners/v4/episode/10024785/screencap/6518d86b9e6ce.jpg</thumb>
  <watched>false</watched>
  <fileinfo>
    <streamdetails>
      <video>
        <aspect>1.7777778</aspect>
        <bitrate>0</bitrate>
        <codec>x264</codec>
        <framerate>23.976023976023976023976023976</framerate>
        <height>1080</height>
        <scantype>Progressive</scantype>
        <width>1920</width>
        <duration>23.847666666666665</duration>
        <durationinseconds>1431</durationinseconds>
      </video>
      <audio>
        <bitrate>0</bitrate>
        <channels>2</channels>
        <codec>AAC</codec>
        <language>jpn</language>
      </audio>
      <subtitle>
        <language>eng</language>
      </subtitle>
    </streamdetails>
  </fileinfo>
</episodedetails>

I'm still using Sonarr to create the metadata file, so in the Jellyfin anime library I don't have any metadata downloader/metadata saver (Nfo) selected.

SolarWolf-Code commented 9 months ago

@vosmiic Use mention you suggestion using AniDB as the metadata provider, do you happen to know how that would work out with Sonarr?

As far as I know, Sonarr will treat these shows as a single entry because they use TVDB as their metadata provider. But AniDB would create different show entries for such as AoT (I believe it's something like 8 different entries). Any ideas? I'm more than willing to test with my selection of shows if I can get a naming scheme work :))

KaXaSA commented 9 months ago

The idea would probably be:

Sonarr

Disable the Metadata file creation (Settings ⇾ Metadata)

Jellyfin

Add/Enable the AniDB metadata plugin
Anime library ⇾ manage library: - enable only 'AniDB' as your 'metadata downloaders' (all of them) - enable 'Nfo' in Metadata savers (not sure if necessary)
Then go back and do a library scan using the replacing all metadata option.

With that, I expect:

Sonarr should probably still do its folder/file management magic, renaming stuff as intended (using TVDB)
Jellyfin should completely replace all the metadata files using the AniDB as the provider, and it should display the correct episode's information on the client¹
Ani Sync should now work with no issues.

¹ not sure how well the AniDB plugin will deal with the TVDB folder, file name... format.

I actually want to test this, but like I said, I think I already tried this before, and there was some issue with the AniDB plugin, but I don't remember what was the issue 🤡, I may still give this a shot on the weekend, not sure yet.

vosmiic commented 9 months ago

I did a quick test on Gintama by only having TVDB metadata and unfortunately it still didn't work.

Thats a shame, but thanks for testing it out.

I have tested using only the AniDB and it still isn't working unfortunately, so I wouldn't suggest @KaXaSA changing your metadata settings just yet. Don't want to accidentally cause your library to break.

@vosmiic Use mention you suggestion using AniDB as the metadata provider, do you happen to know how that would work out with Sonarr?

I'm not sure honestly, in my development environment I don't use Sonarr so its hard to tell on my side. I would presume Jellyfin will be able to map to the AniDB format even if Sonarr transforms the files and folders into the TVDB format, but I would probably hold off since I don't actually think it would fix anything as stated above.

I will attempt to implement the potential solution outlined in my previous comment and let you guys know how I get on.

vosmiic commented 9 months ago

So I am able to calculate the absolute episode number (unfortunately the Jellyfin API does not provide it) by simply iterating through the seasons and getting a count of the total episodes up to the one being watched, which we can use to compare to the offset and get the correct AniDB ID. The only issue with this method is that it relies on the series being complete. The absolute episode number will be incorrect otherwise. For example if you only had season 1, 2, 3 and 7 of Gintama (according to TVDB) and watched an episode in season 7, the plugin would only count the episodes in season 1, 2 and 3 to get the offset because they are the only seasons available to the plugin. Which would probably give you an absolute episode number of an episode in season 4. I'm thinking this is probably a niche issue as I would presume most users would keep a full library, but its enough to make me wonder if there is a better way of approaching this.

SolarWolf-Code commented 9 months ago

@vosmiic There definitely is a better way. And I'm working on a Python script that I'll send here and you can try to translate it to C#. I'm almost done, but I think we have all the info we need

wilyub commented 9 months ago

I've reviewed the logic changes you suggested @vosmiic and I'd like to throw my hat in the ring with another suggestion (some psuedo-code for attributes because I don't know all the correct names for them). Let me know your thoughts.

if (foundAnime.Count() == 1) return int.TryParse(foundAnime.First().Anidbid, out aniDbId) ? (aniDbId, null) : (null, null); 

if (watched_anime.season == 1) {//If the season is #1, then we can go off of the AniDB since all AniDB entries are 1 season. This is the old logic copy pasted since it works very well for anidb/season 1 entries.
    for (var i = 0; i < foundAnime.Count; i++) { 
        if ((int.TryParse(foundAnime[i].Episodeoffset, out int episodeOffset) && episodeOffset <= episodeNumber) || i == 0) { 
            if (foundAnime.ElementAtOrDefault(i + 1) != null && int.TryParse(foundAnime[i + 1].Episodeoffset, out int nextEpisodeOffset) && nextEpisodeOffset <= episodeNumber) continue; 
            logger.LogInformation($"(Tvdb) Anime {foundAnime[i].Name} found in anime XML file (using Tvdb ID)"); 
            return int.TryParse(foundAnime[i].Anidbid, out aniDbId) ? (aniDbId, episodeOffset) : (null, null); 
        } 
    }   
}
else {//If the season number is not 1, then we must have TVDB-esque seasons (something like Gintama Season 4). We can now use this logic specifically for the TVDB-season edge cases like Gintama/Bleach/Etc...
    for (var i = 0; i < foundAnime.Count; i++) {
        if (foundAnime[i].defaulttvdbseason == 'a') { 
            //I noticed that some "a" shows like Detective Conan and One Piece do not have offsets in the mapping-list.
            //For those shows it might be best to just have the numbering be absolute (so instead of S20E20 just have S01E500 or whatever it is).
            anime_mapping_list = foundAnime[i].mapping-list; //Assume this is in an array-like structure
            for (var j = 0; j < anime_mapping_list.Count; j++) {
                if (anime_mapping_list[j].tvdbseason == watched_anime.season) { //EX: Gintama Season 4 Episode 3 -> 3 - (-150) = Gintama (AniDB Season 1) Episode 153
                    return episode_number_to_send_to_provider = watched_anime.episode_number - anime_mapping_list[j].offset 
                }
            }
        }
        else { //Defaulttvdbseason is a number
            if (foundAnime[i].defaulttvdbseason == watched_anime.season) { //Found a match in the season numbers
                return anidb_id_to_send_to_provider = foundAnime[i].anidbid // So Gintama Season 5 Episode 10 should send ID: 8126 (Gintama Seasons 1-4 are ID: 3468)
                return episode_number_to_send_to_provider = watched_anime.episode_number // So Gintama Season 5 Episode 10 should send number 10
                // Obviously you can't have two return statements, just return both quantities.
            }
        }

    }

}

wilyub commented 9 months ago

Good catch! Didn't think of that. Any ideas on shows like One Piece/Detective Conan that don't have seasons in the mapping list? Personally I have Detective Conan numbered in absolute episodes, but I can see some people already having collections in TVDB format that would be troubled by this. Outside of writing the mappings ourselves I'm not sure how to approach this one.

vosmiic commented 9 months ago

Any ideas on shows like One Piece/Detective Conan that don't have seasons in the mapping list?

Hmm I'm not sure. Unfortunately there may be some shows that we simply will not be able to support. I guess the best course of action would be to implement a solution for the majority of situations and then investigate if we can support the niche cases.

KaXaSA commented 9 months ago

What if you had a plugin option where the user could pick between using TVDB OR AniDB? Would that be an easier said than done situation, or are there other reasons for not going that route?

wilyub commented 9 months ago

This is great to hear, I think if it's only that one hick up we can try to get that fixed on the database side by chatting with the arm repo people or anidb people. Thanks for working through this :)

vosmiic commented 9 months ago

So development work is mostly complete, the changes have not yet been merged to master though. Wondered if anyone was able to do some basic testing using their library to see if it is working for them, I have tested in my limited development library and all seems to be working but obviously more testing is always welcome and would be much appreciated, but obviously don't feel pressured to do so.

It should be as stable as the current release version of the plugin in terms of not ruining your Jellyfin environment, but since I have changed the areas that detect what anime you are watching it might be best to keep an eye on the log to make sure it hasn't updated something incorrectly.

Also feel free to use any combination of metadata providers, hopefully what I have implemented means that it isn't dependent on specific providers.

Here is the dll file (its in a zip because GitHub won't let me upload a raw dll file): jellyfin-ani-sync.zip

Or if you would prefer, you can build off of this branch: https://github.com/vosmiic/jellyfin-ani-sync/tree/anime-detection-changes

Thanks.

wilyub commented 9 months ago

Thanks for the update! I went ahead and tested it on an episode of Aggretsuko Season 5 Episode 7 to see if the regular functionality still works. It looks like something may have broken? (Regular version of plugin has handled all my Aggretsuko S5 updates with no troubles). Here are the logs.

2023-12-06 20:58:20.092 -05:00] [INF] [3] jellyfin_ani_sync.UpdateProviderStatus: (Tvdb) Anime reference found in anime list XML [2023-12-06 20:58:20.093 -05:00] [INF] [3] jellyfin_ani_sync.UpdateProviderStatus: (Tvdb) Anime Rage Mission found in anime XML file [2023-12-06 20:58:20.094 -05:00] [INF] [3] jellyfin_ani_sync.UpdateProviderStatus: Retrieving provider IDs from offline database... [2023-12-06 20:58:21.387 -05:00] [INF] [30] jellyfin_ani_sync.UpdateProviderStatus: Retrieved provider IDs [2023-12-06 20:58:21.389 -05:00] [INF] [30] jellyfin_ani_sync.UpdateProviderStatus: Using provider AniList... [2023-12-06 20:58:21.556 -05:00] [INF] [30] jellyfin_ani_sync.UpdateProviderStatus: User does not have plan to watch only ticked [2023-12-06 20:58:21.558 -05:00] [INF] [30] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (Aggretsuko) found on completed list, setting as re-watching [2023-12-06 20:58:21.563 -05:00] [INF] [30] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (Aggretsuko) has already been watched, marking anime as re-watching; progress of 7 [2023-12-06 20:58:21.779 -05:00] [INF] [30] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (Aggretsuko) found on Completed list [2023-12-06 20:58:21.780 -05:00] [INF] [30] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Series (Aggretsuko) found on Completed list, but user does not want to automatically set as rewatching. Skipping

The plugin set my Aggretsuko Season 1 as Rewatching and marked episode 7 as rewatched (despite what the logs might say). It looks like it's trying to find the name of the anime (from TVDB) based on the name of the episode? (Episode name for S5E7 is Rage Mission). I'll refrain from testing other stuff while this gets sorted out. Thanks again for the help.

vosmiic commented 9 months ago

Thanks for testing it @wilyub

It looks like it's trying to find the name of the anime (from TVDB) based on the name of the episode?

That's just an issue with the logger, I think its looking up the correct entry by the TVDB ID.

How many seasons do you currently have in your Jellyfin library for Aggretsuko? It looks like its finding just a single season, so I presume you only have season 5?

I'll look into the issue. Will have to put this down for a bit soon though, going to be away from my dev PC for a few weeks over the Christmas period so this probably won't be ready for release until sometime in the new year.

wilyub commented 9 months ago

Thanks for the comments, no worries about release window. Enjoy the holidays!

You are right that I only have Aggretsuko S5 in my library. This is not the usual case with my stuff, so I'll go ahead and test this out on some other shows I have that have the full library present. Will send an update later on my results.

KaXaSA commented 9 months ago

I just replaced the plugins\Ani-Sync_2.9.0.0\jellyfin-ani-sync.dll with the one in the zip file without changing anything else, not sure if that's ok.

Tested with: One Piece (1999) - S21E188 - 1079 - Episode Title.mkv

[22:32:07] [INF] [86] jellyfin_ani_sync.UpdateProviderStatus: (Tvdb) Anime reference found in anime list XML
[22:32:07] [WRN] [86] jellyfin_ani_sync.UpdateProviderStatus: (AniDb) Could not lookup using absolute episode number, falling back to season number lookup
[22:32:07] [INF] [86] jellyfin_ani_sync.UpdateProviderStatus: (Tvdb) Anime One Piece could not found in anime XML file; falling back to other metadata providers if available...
[22:32:07] [INF] [86] jellyfin_ani_sync.UpdateProviderStatus: Using provider AniList...
[22:32:07] [INF] [83] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Found matching series: ONE PIECE
[22:32:07] [INF] [83] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Attempting to get season 1...
[22:32:07] [INF] [83] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Could not find any related anime
[22:32:07] [WRN] [83] jellyfin_ani_sync.UpdateProviderStatus: (AniList) Could not find next season

How many seasons do you currently have in your Jellyfin library...

I also just keep a few episodes in Jellyfin, having all seasons is pretty rare.

( Happy Holidays 🥳)

vosmiic commented 9 months ago

You are right that I only have Aggretsuko S5 in my library.

This should be sorted now, basically wasn't handling the case of not having the entire series in the library.

As for the issues you are seeing @KaXaSA, what metadata do you have on your One Piece entry in Jellyfin? What AniDB ID and/or TVDB ID does the series have? Odd that it isn't able to do a simple ID match.

Thanks both for testing, will upload a new dll file with the fixes once I have confirmed its sorted out all of the found issues.

KaXaSA commented 9 months ago

what metadata do you have on your One Piece entry in Jellyfin? What AniDB ID and/or TVDB ID does the series have? Odd that it isn't able to do a simple ID match.

(removed unnecessary metadata info)

Serie metadata | One Piece (1999)

...
  <title>One Piece</title>
  <id>81797</id>
  <uniqueid type="tvdb" default="true">81797</uniqueid>
  <uniqueid type="imdb">tt0388629</uniqueid>
...

Episode metadata | S21E188 - 1079

...
<season>21</season>
  <episode>188</episode>
  <uniqueid type="tvdb" default="true">10073293</uniqueid>
  <uniqueid type="sonarr">2447</uniqueid>
...

I checked the metadata in Jellyfin too, just to make sure, and it matches these .nfo files.

vosmiic commented 8 months ago

Thanks @KaXaSA, back to looking at this. Will hopefully have some improvements out soon, I'll keep you guys updated.

vosmiic commented 8 months ago

So I have created a new version that should sort out the issues of not having the entire show in your library. As for the issues with Aggretsuko @wilyub, I can see that it won't recognise season 5 because its not in the XML file yet. I tested it with season 4 and it seems to be working. And @KaXaSA I think the way the plugin currently works it won't find One Piece because it can't calculate the offset as it hasn't got all of the episodes, and it can't fall back to looking up by season number because the XML file only has a single season but multiple entries in the XML file (the specials). I'll investigate if there are other ways of still reliably detecting the anime in the XML file in these cases.

vosmiic commented 8 months ago

Looked a bit more into it, read a bit more into how the anime list XML is formatted and I don't think its possible to map to a show that is formatted in Jellyfin in the aired order and is not complete. I think the XML can be a bit confusing sometimes, as some entries contain aired order and others are absolute. Perhaps adding a clause to the metadata comparability page stating that the plugin works best at detecting shows when the series is fully present in Jellyfin.

wilyub commented 8 months ago

Thank you for the hard work! I think the full show being present is a reasonable thing to put here (since that's the best we can get with the current XML resources).

vosmiic commented 8 months ago

So these changes are now in a beta release, which can be added to your local environment by adding the new beta manifest. Like the readme states, both repos can be used at the same time, and any new stable releases will replace the beta releases and vice versa when they become available.

Don't worry about participating in the beta release if you aren't comfortable with it, if no issues are found it will be promoted to stable in a few weeks once a decent amount of testing has been done. Going to close the PR but it can be re-opened if issues related to it are found. Thanks

vosmiic / jellyfin-ani-sync