search

ZeroQI / Absolute-Series-Scanner

Seasons, absolute mode, Subfolders...
1.01k stars 155 forks source link

Movie files in root folder detection issue + COGIMA (movies+series in Series library) matching issue #299

Closed mufunyo closed 3 years ago

mufunyo commented 4 years ago

So if I go to thetvdb.com and enter Paris is Burning, I correctly get this entry: https://thetvdb.com/movies/paris-is-burning

But if I use Absolute Series Scanner to try to match "Paris is Burning (1990)", I only get "The Bronx is Burning (2007)" as a match result.

I have Absolute Series Scanner set up to work with Colima for a combined series/movies documentaries library, but even if I disable Colima and set the library to scan using TMDB or TheTVDB it still exhibits this behaviour, so I'm assuming the bug is in Absolute Series Scanner.

I tried updating to the latest version but that makes no difference.

NB: title matching has always been extremely fragile, having any sort of group name in the filename always causes the match to completely break and show up as "Season 502". It's a pain to have to rename everything to a perfect scheme but I guess I'm sort of used to that with Plex by now.

ZeroQI commented 4 years ago

The way you named your movie in a MOVIE library would work no problem had you named it for the metadata source used (TheMovieDB with COGIMA with movies, NOT TheTVDB)...

However you are putting a MOVIE in a SERIES library (which is not supported by Plex or ANY other ecosystem but i hacked my way around), ignoring the scanner read-me that states "Movies in series libraries support (names same as folder or contain " - Complete Movie")", and without including the scanner logs that would say "no series title, contact the developer" had you looked at them first, which we both know you didn't (in Plex logs folder, subfolder with the library name)

You don't need a perfect scheme but proper foldername. In the setting of a series library with movies, this are the supported schemes:

Additionally, using a third party agent (https://forums.plex.tv/t/rel-combined-library-metadata-agent-colima/210673) i do not support, which states:

Colima will look for a unique identificator to distinguish movies from tv shows. The agent pulls data from TheMovieDB for all movies it can find this way.

In TheMovieDB the title is "Paris Is Burning (1991)". The matching is done by the AGENT, which uses TheMovieDB for movies naming, as TheTVDB is for TV Series, although it started supporting movies, it is nowhere close to the number of entries in TheMovieDB nor supported by Colima currently...

If added propertly by the scanner as s01e01, try to match using "Paris Is Burning (1991)". If it works, please donate as you make me lose my time for an unsupported issue, providing no logs. This was documented in the readme as exhibit A:

Creating a post or bug report without including relevant information like logs, serie impacted, or having not said the troubleshooting steps were followed will be recommended to pay the RTFM tax by donating. This part is called 'Exhibit A'.

mufunyo commented 4 years ago

You don't need a perfect scheme but proper foldername. In the setting of a series library with movies, this are the supported schemes:

  • Paris is Burning (1990)/Paris is Burning (1990).mp4

That is exactly what I did except the extension is mkv.

In TheMovieDB the title is "Paris Is Burning (1991)".

This seems to be the source of the issue, as changing the filename to 1991 fixes the match. I still don't at all understand how "Paris is Burning" (no year) or "Paris is Burning (1990)" yields NO match at all to the 1991 title, while instead it "finds" a title that is completely unrelated apart from having "is burning" in the title, not even being released in the same decade. At this point I've been gently massaging filenames and metadata to keep Plex and its scanners happy so I hope you can understand some of the frustration when such a completely illogical match is thrown up when the more obvious one is not even available for manual selection. With Plex's built-in scanners it is at least possible to just enter the id number of the tvdb/tmdb entry, but this also doesn't work with Absolute Series Scanner.

Creating a post or bug report without including relevant information like logs, serie impacted, or having not said the troubleshooting steps were followed will be recommended to pay the RTFM tax by donating. This part is called 'Exhibit A'.

I would recommend making an issue template instead of becoming angry at me for not reading the manual. An issue template acts as a catching net for people like myself who missed the relevant paragraph in the readme, to provide instructions on filing a proper issue report. I can set one up for you if you want, to express my thanks for your (albeit extremely cranky) solution to the problem.

ZeroQI commented 4 years ago

Scanner

With Plex's built-in scanners it is at least possible to just enter the id number of the tvdb/tmdb entry

IF and only if it works with the same agent and another scanner (plex series scanner default one), then let me know the naming used and ideally the scanner logs when using my scanner so i can make it work.

HAMA the agent i maintain does accept tvdbid directly as well as forced ids in the shape [tvdb-xxxxxx], [anidb-xxxx], etc...

I am cranky because i believe you create a scanner report for an issue regarding a third party agent, despite the very initial description detailing what the scanner actually do:

If a file is not showing in plex or showing at the wrong season and/or episode number, or is not passing through the forced id, then it is a scanner issue. Anything else is metadata related and agent specific.

Agent

The agent is the one using the title to assign a unique id (if match is >80% accurate as given by the agent search function), which the update function of the agent uses to download the metadata...

Your Plex COGIMA metadata agent matching (search function) has the following issues:

Different databases can be off by one year, although without (or even with the wrong year) the year it should have been able to match to be honest, so it could deserve a bug report at https://github.com/defract/Colima.bundle/issues but the code is 3 years old and no answer to the bug reports from june 7th and 11th so i do not expect you to get an answer...

If you would be so kind to create a template, as i believe it would be of assistance and improve my volatile mood and quality of bug reports reported, i would approve the change straight away

mufunyo commented 4 years ago

I'm very sorry about being so impatient. I thought I skimmed the readme but reading it properly all the mistakes/assumptions I made could have been prevented. I did initially suspect Colima, but like you said the plugin is dead (and it is the only one of its kind as far as I know), so I did minimal testing to rule out the agent as the problem (by using TheTVDB as agent, which yielded the same result - but that is probably because the TheTVDB agent is not made to handle movies).

Different databases can be off by one year, although without (or even with the wrong year) the year it should have been able to match to be honest, so it could deserve a bug report at https://github.com/defract/Colima.bundle/issues but the code is 3 years old and no answer to the bug reports from june 7th and 11th so i do not expect you to get an answer...

It should really have matched, and part of my frustration is that Plex is glacial in acknowledging/fixing issues and refuses to implement mixed libraries we are left to hack our way into a functioning database. Whenever I encounter issues I find Plex forum results that are 8+ years old with no acceptable solution. As my on-disk format does not always match the kind of perfect naming schemes that Plex likes (I want to preserve the original release names with no foreign files added), I have to use overlay file systems and symlinks to make the original filenames ignored by plex using .plexignore files, while providing a link that uses the right format, all while leaving the original folders untouched.

If you would be so kind to create a template, as i believe it would be of assistance and improve my volatile mood and quality of bug reports reported, i would approve the change straight away

I've created the pull request and included the scanner/agent distinction as a warning as well as an instruction to attach a log file. I hope this offsets the frustration I've caused you.

ZeroQI commented 4 years ago

Thanks for acknowledging, i did at some point modify the readme to prevent such occurences and stopped when mot didn't read it...

It should have matches and would need correcting to fix that

In the works of metadata gathering, it is very inflexible, Emby, xbmc/Kodi, Plex... movies or series libraries, series folder at root only (although i managed to painfully hack my way around with grouping folders using recursion)

Agents are anidb or moviedb or tvdb, but pretty rigid

I did the scanner to allow for anime movies and series in the same library as it is pretty much a requirement for Anime, but it also is for documentaries... Movies however need their own folder or " - complete movie" but i am considering "title (year).ext" as it could help users although i do not want them to dump random files at root to after complain so i am split over the decision...

normally, if you name your folders well, you don't have to rename the files... That's what i made the scanner for... Your way seem so convoluted, if you give me few exemples of non supported exemples, can see if there is a way without renaming the file...

mufunyo commented 4 years ago

In all fairness the Plex default scanners do an okay job at matching movies and regular episodes in their native file/folder naming. The series scanner/agent just completely breaks down with specials as these are released as for example S01E00 while TheTVDB expects S00E01 for specials. Also multipart miniseries don't work because the releases are called Some.Miniseries.Part1, Some.Miniseries.Part2 and the Plex scanner just doesn't understand it so it marks them as "Season 0 Episode 502" which you can't edit or correct manually aside from using a different filename. I never understood where this number 502 comes from, I tried searching for it on Google but didn't find anything.

Standard release naming keeping all file and folder names intact would be as follows: Some.Series.S01E01.1080p.BluRay.x264-GROUPNAME/grpname-someseries-s01e01-1080p.mkv The filename itself can be almost anything, sometimes it's a copy of the release name (same as the parent folder), sometimes it's a cryptic shortening in all lowercase. Another thing Plex can't deal with is two episodes in one airing, which will be denoted as follows: Some.Series.S01E02E03.1080p.BluRay.x264-GROUPNAME If you're lucky the TheTVDB entry will have separate episodes for both so that it just treats the file as episode 02 and in the library it will appear as though episode 3 is missing. If you're unlucky the TheTVDB entry will treat the double episode as a single episode (for instance the first episode of Firefly) so all the remaining episodes will be numbered off by one compared to the release numbering. Between the movie name or episode number and the resolution (1080p) there can also be keywords like PROPER, REMASTERED, EXTENDED, REAL, or even a combination like REAL.REAL.PROPER.

Also releases are packed in RAR files which Plex has never supported so I use FUSE filesystems to work around Plex's shortcomings. I use rar2fs to present the releases as unpacked mkv files even though on disk they are untouched rars. Then I have a mostly empty folder tree with plexignore files and symlinks which I overlay on top of the rar2fs volume using mhddfs to ignore the original release filenames of things Plex cannot match, and have a symlink with a properly formatted filename only for those specific episodes (like specials).

For the ASS/Colima library where I have my documentaries, I have to plexignore the original files in all cases because the release names are too complex.

ZeroQI commented 4 years ago

I understand better for the compressed videos, but this seems more complex than decompressing all in another folder to allow to continue to share the original file... If the size becomes a problem then the amount of data is maybe too big... Because of torrents, i keep a download folder, copy and rename to the proper location then delete when it is shared...

For movie libraries with tmdb agent it work pretty well to be fair.

For the tvdb specials numbering well you do have to follow some convention... 501+ comes from anidb numbering convention so it does not conflict with existing metadata... I use to skip such files but having then added this way allow users to see them. Would you rather have them not showing at all?

Part1 etc should be detected by the scanner property as s01e01, if not I need the scanner logs to see if fixable.

Double eps will be entered as two separate entries Each showing the correct metadata, so what do you complain about? If TheTVDB have separate entries but the release group do group them, then they are being dicks... If doesn't work need scanner logs...

If the release group make a mess of the filename, then I think you should blame them and not the scanner... Have you seen the code in the scanner already? Far from simple to try and handle most naming conventions... You can always document which file naming cause problem you feel should be supported, but need scanner logs...

Yes to fix numbering you need to rename files. To fix matching you need to edit the folder name unless you put the tvdb ID in a file called tvdb.id in the series folder... It will replace the title with thetvdbid which TheTVDB agent supports...

Wouldn't group movies with series if you can avoid but anime can't, documentaries can't neither, but other cases can I believe...

I cannot understand why you need a movie like this one in a tv series library to be honest... If something is worth watching and keeping, it is worth renaming in my opinion...

My scanner dies a better job at series than Plex default one (otherwise we'll don't use it?), I can work on each shortcoming separately with you but:

Do you have actual naming conventions that should be supported and are not?

mufunyo commented 4 years ago

I see you edited your post but I want to make clear that I don't mean to bitch at you. You were curious about my setup and edge cases that weren't supported so I was just making conversation by telling you what I've been doing to keep things organised and working within the confines of what is supported in Plex.

I understand better for the compressed videos, but this seems more complex than decompressing all in another folder to allow to continue to share the original file... If the size becomes a problem then the amount of data is maybe too big...

With unlimited Google Drive size doesn't become a problem but rar2fs has like 1% cpu overhead since the rars don't actually use compression so I would consider that part of the chain the least of a problem. I think it's a bit silly that Plex refuses to implement it natively, just like ISO file support, but it's their product, their choice. We can leave whenever we want to.

For the tvdb specials numbering well you do have to follow some convention...

I think a .plexoverride file that maps filenames to different metadata (like changing episode numbers) could be a very elegant solution. I think I'll suggest it over on the Plex forums. This way naming format and numbering convention doesn't rely on renaming actual physical files. Some people might be using read only filesystems and can't rename things, or like me want to preserve original releases.

If TheTVDB have separate entries but the release group do group them, then they are being dicks...

The release groups follow historical convention. In the case of Firefly, the releases that dictated the episode numbering came out in 2006, and TheTVDB.com did not even exist until 2007. Some people want to "correct" numbering after the fact but the real world is messy. It's kind of like the people that wanted to change the meaning of kilobyte from 1024 bytes to 1000 bytes. The people that decided on 1024 bytes back in 1960 or whenever it was weren't trying to be dicks, that just seemed the best at the time. Trying to force change just makes it more confusing for everybody.

  • Plex sucks less than other solution with no custom scanner and little agents, and you can with Plex pass submit change request and have people vote for it, and it allows custom scanners and agent but far from being exempt of defaults

This has been my conclusion as well, which is why I'm still using Plex and not another solution. It's good that people like you are out there making the product better to use for everybody.

  • you want weird shit supported? You can always pay for features like the guy that commissioned Lambda agent as he wanted to not lose all the customization he did over the years...

I think that it might come to that at some point, yeah. As the libraries grow larger it could be a good investment to have things work automatically instead of all this manual work. We'll see :)

If you're tired of this discussion we don't have to go on. It's been very informative and insightful to me though, so thank you for taking the time to explain things to me.

ZeroQI commented 4 years ago

I did indeed edit as writing in the middle of the night when cranky is a bad idea

I would like Plex to indeed support

TheTVDB follow the source provider info which sometimes makes no sense. two separate episodes seen as double ep, etc... I did use sonar as you can select all eps and rename following a convention and delete the files from temp folder, very practical and i believe it's the system most used by users...

Oh yeah hated HDD manufacturer for 1k=1000, or powerline 600Mbps for 300Mbps link (up and down)...

I am glad you feel that way, although cranky i do like talking if meaningful... I try to make it better but at times can't even read my own code back (few peoples contribute and helped as well)

Am not saying i would code anything as my time runs short, but i would add non-weird possible scanner improvements:

I believe that would raise the use case, but dunno when i will have the motivation to do it...

ZeroQI commented 3 years ago

support "title (year).ext" at root without " - Complete Movie" is a good idea, adding to master code

ZeroQI commented 3 years ago

@mufunyo Apologies for the delay, no time and too many bug reports on ASS, Hama, YouTube, Lambda and lost motivation...

I can add movies in iso/rar/zip format but the video won't play... Please test and report if "Movie title (year).ext" format at root fails, closing in the meantime