Finding a way to allow "grouping folder" again

[dgw]

I appreciate that there are technical barriers to having the scanner recognize paths like \LibRoot\Show Title\[subgroup] Show Title [releasetype][resolution]\[subgroup] Show Title - nn [format][info] [CRC32].mkv, but I consider it a pretty big usability issue that this isn't supported. It unnecessarily dictates how the user must organize their files to even be detected and pulled into Plex.

The workarounds:

• Store batch folders etc. outside the library root, and hard-/symlink to the files from inside the library. (Potentially automated with some scripting in download client/post-processing, but kind of ugly.)
• Add each "grouping folder" to the Plex library as shows are added. (Not possible to automate, AFAIK, and leads to a very cluttered list of library folders.)
• Store batch folders directly in the Plex library root. (Very ugly; gives [subgroup] Show Title [Batch][720p] folders alongside Show Title, and requires putting multiple releases of the same show alongside each other in the library root.)
• Extract the files from batch folders and dump them straight into \LibRoot\Show Title\. (May require more work to track down the source of a given file if necessary, and breaks the ability to continue seeding in place unless hardlinks/symlinks are used—see workaround 1.)

Frankly, the way ASS handles directories these days has led to me watching much less anime, simply because I find it annoying to organize things the way ASS demands. Could I go back to Plex's built-in scanners? Sure, but those have other, more glaring issues with anime. It's somewhat ironic that Plex's built-in series scanner handles just about any directory structure I can throw at it, but ASS handles anime way better at the expense of rigid enforcement of library file structure.

Ultimately, I guess I just don't understand why ASS can't match more things. For times when it matches files it shouldn't, we have .plexignore files (which I used extensively a couple years ago for excluding the OP/ED/etc. from BD batches). @ZeroQI, are you absolutely set on the current way of doing things, or would you be open to making ASS more liberal in what it detects (whoever actually writes the code)?

[ZeroQI]

I wrote ASS from scratch. I do not force people to use it and am not adding restrictions not present in the original plex scanner. I made it the most liberal possible without compromising fonctionalities and added quite some that are in no other scanner, logging, etc...

I do believe Plex scanner does not really handles grouping folders. Last yime i checked, it created one entry for the grouping folder and set every series episodes as duplicate episode files... Since you have the same serie in grouping filders it actually work for you but this is the first time i see such arrangement. Why store multiple release? If that is no longer the case let me know.

Why not using the old scanner? That will rescan every time and corrupt your plex database once in a while (every 2 weeks when coding)...

The new one skips grouping folders logging to add grouping folder as root folder.

Plex requires to use as root folder the series parent folder. You would have a mess in one single serie had you grouped all dragon ball series in a grouping folder ... I considered editing the library to had grouping folders but that means the scanner write outside its logging folders which could have unforeseen effect and security liability issues...

Feel free to submit changes but do not write outside log and agent folders, and do not break it for users not grouping releases together.

PS: using the ark folder functionality you could bypass some folders as if they didn't exist. Find me a pattern for the naming without making me add multiple release group and i would add it back... ''['release'] [group] Serie name'' for ex?. I need to rule out grouping folders unless they are grouping the same derie and i need a way to tell them apart by folder naming...

Edit: "handles just about any directory structure I can throw at it, ". no it handles just grouping folders with the exact same series inside, which only helps you. I literally so far never had a listing sent (the scanner create filelist to aid t/s) with multiple release of the same series...

I only implement Plex restrictions i try not to restrict further and believe i made it better for anime. You can try BABS scanner that does absolute numbering support but i do believe mine is better. I will improve anything but not if it make the experience worse for normal users. Could keep an old version of scanner for people needing grouping folders more actively...

[ZeroQI]

It doesn't make sense to me to have multiple release of the same anime in the same library when Plex allow trancoding and even to select the bandwith when doing so...

You are the only person that ever was in this situation that therefore give you a unique predicament If it is that important to have multiple release then add grouping folder as root folder or use an older version should be doable...

Latest version supporting grouping folders is from the 15th of November 2015 https://github.com/ZeroQI/Absolute-Series-Scanner/blob/3d059940580065d6896b3c4d677401a862084d70/Scanners/Series/Absolute%20Series%20Scanner.py

It also support plexignore files which i believe are not supported with the scanner currently (please confirm if you know). Please try to see if it solve things for you but tvdb4/5 and anidb2 mode are not implemented, and let me know

[dgw]

It doesn't make sense to me to have multiple release of the same anime in the same library when Plex allow trancoding

It does if your server doesn't have the horsepower to transcode without buffering every five minutes.

You are the only person that ever was in this situation

I very much doubt that. Just because I'm the only one talking about it does not mean I'm the only one to experience it.

Ultimately, maintaining multiple releases is only part of it. Even shows for which I keep only a single release (the majority) are easier to work with from the file manager if kept in a grouping folder that is just the series name. If nothing else, it helps with finding shows quickly when something needs updating/replacing, because the file alphabetical sort works best without [tags] leading the way. It also looks much better than the circus of release group & format tags I get by dumping anime batches straight into the library root, as ASS currently requires.

You are correct, of course, that you're not forcing anyone to use ASS. I use it because it was a great way to scan anime into Plex. It still is—it just takes a bit more work than before because of the directory structure requirements (which took me by surprise upon upgrading—obviously my fault for not reading the documentation closely enough before replacing the bundle). For some reason I recall not liking BABS, but it has been a few years so perhaps I should create a test library just to see.

Anyway. The whole point of this suggestion is to make ASS easier to use. I made it because I want to see ASS improve. If you don't want to code it, that's fine. Being the maintainer of an open source project doesn't mean you have to agree with feedback users like me give you, and it certainly doesn't mean you're required to implement our ideas. Tag the issue "help wanted", or close it if you like.

For what it's worth, I would like to learn more about Plex scanners at some point. This would probably be a good project to go along with that, and that's why I asked if you would be open to bringing support for grouping folders back. It sounds like you are. Feel free to even assign this issue to me, so it shows up in my GitHub issue dashboard and reminds me to do the research needed to be able to work on it. I don't like to be just a whining user; if rolling up my sleeves and learning how to write scanners is what it takes, I should do just that.

[ZeroQI]

Fair point you are the first to report... brackets are dropped so they do not bother the scanner BABS was lacking, which is why i did my scanner, and it was slower and created corruption, but scanned all during the root scan, and could add anything including grouping folders, but had to rescan everything every single time, when untouched folders are normally skipped so i changed the workings to use Plex caching but in your case with the grouping folder properly labeled and same serie within, it actually works decently with plex scanner (not for absolute numbering though). wouldn't work with a dragon ball folder and all subseries within... The only way i can see to do both is to maintain 2 versions of the scanner. I can add a second scanner file, with the latest grouping folder and .plexignore, then bring it to level feature wise... I would not recommend it first, only if many grouping folders or plexignore support needed as it doesn't use usual caching and i believe it can corrupt Plex database

Be carefull, scanner are a pain to devellop, due to lack of logging (which is why i developped my custom scanner logging) but due to caching, even if you change the behaviour, if no files change in the episodes folder, they are not reflected, lost countless hours due to that... That and fixing one thing to break another...

You were very whining but took time to document and while foreign to me i can see the benefit of keeping both method as a scanner learning tool and as functionalities differ fairly now

I have created https://github.com/ZeroQI/Absolute-Series-Scanner/blob/master/Scanners/Series/Absolute%20Series%20Scanner%20(legacy).py

Please try this legacy scanner and confirm if any issues Will need then to use generic shorted function to avoid list of logs folders per os and add tvdb modes created since...

[dgw]

OK, this is on my list of things to look at. Thanks for the invite to collaborate. Sounds like I should plan to use a test library and create/delete as necessary to get around the scan caching during testing (when I reach that point).

Before I dig into the code, is there any proper documentation on developing scanners I should read? Or is it pretty much "read scanner code to figure out how scanners work"? Kinda frustrating how little documentation on plugins of any kind I've been able to find, really. One could reasonably expect better documentation from Plex.

At any rate, my most productive first steps code-wise are probably to read the code for Plex's series scanner, the code for the current version of ASS, and the legacy version you added earlier. Am I correct that the code concerning which folders and files are included should be confined to Scan()? It appears at a glance that all other functions defined in ASS are helpers of some kind, not called directly by Plex itself.

Optimistically, I'd like to believe there's a way to support grouping folders in the current release, but I may learn better as the rabbit hole gets deeper… Having two different versions of the scanner seems awkward, but if it's the only way, it's the only way. If I come up with any ideas I'll bring them up—like processing a grouping folder if it contains subfolders whose names contain substrings that fuzzy-match the parent above a certain confidence, and skipping otherwise? That might not even be practical, don't know enough yet, just speculating.

[ZeroQI]

https://forums.plex.tv/discussion/comment/971278/#Comment_971278 Read framework pdf Read legacy scanner code Scan() is called once for root folder once per subfolders. If you add eps for grouping folder subfolder rom root scan call all possible but will scan all everytime. If you add eps for grouping folder subfolder all will end up in a single serie associated to the grouping folder. This moght have changed but that's how it used to be so to use plex caching so it skips unchanged folders i had no choice but recode and do the current way. Since for me grouping folders were for separate series for same universe like tenchy or dragon ball never thought about multople release since i would only keep the streamable one in plex... there is a project to share trancoding across multiple computers btw on the forum.

I could not fully grasp plex scanner way of doing things functions wise and disliked how it remove characters so i did my own functions and it even supports japanese characters...

I wanted it so people could learn from reading the code, since the framework pdf was limited for both scanner and agents...

The legacy scanner is the easiest mostly as code is all contained and all happens within a single scan() call... The tricky bit is recursive calls for .plexognore. to my knowledge scanner need to support the file Was easy on a single call function...

That should help. Be used to delete and re-create the library everytime you have issues if not adding all on root folder scan call.... If it crashes no much 8nfo on faulty code... it drove me nuts Always keep copy of latest working one and work out difference in notepad++ compare plugin...

That should help. Btw, any metadata agent on cody/xbmc, emby. Media companion require the serie directly in the folder shared even plex but it let hack your way through like i did.

No options possible but did it by detecting presence of file in root or logs folder but managed to remove them over time...

Am seft taught python but did nice functions in hama common.py file to create embricated arrays in one go...

That shoulp help...

[EndOfLine369]

@ZeroQI, I also don't think that legacy should be brought back to life as its also before my time and I fixed quite a few bugs in file processing aside from any corruption issues.

Can't we just change the scanner to not skip when the string "[grouping]" is on a folder? Should potentially just need a change of adding a test. and not "[grouping]" in reverse_path[1] ?

FROM:
    if len(reverse_path)>1 and path.count(os.sep):         #if grouping folders, skip and add them as additionnal folders
      Log.warning("Grouping folder: '%s' skipped, need to be added as root folder if needed" % path)
      Log.info("".ljust(157, '-'))
      return

TO:
    if len(reverse_path)>1 and path.count(os.sep) and not "[grouping]" in reverse_path[1]:         #if grouping folders, skip and add them as additionnal folders
      Log.warning("Grouping folder: '%s' skipped, need to be added as root folder if needed" % path)
      Log.info("".ljust(157, '-'))
      return

[ZeroQI]

Legacy does support .plexignore and doesn't need to add root folders. i wanted to bring it back to standards but that said, this suggestion would fix his problem, and only because all his releases in grouping folder are the same serie so "[multi-releases]" in foldername or single subfolder should not be skipped. Good suggestion :D

if not match and len(reverse_path)>1 and path.count(os.sep) and len(subdirs)>1 and not "[grouping]" in reverse_path[1] and not "[multi-releases]" in reverse_path[0]: #if grouping folders, skip.User to add them as additional folders

either grouping folder has "[multi-releases]" or subfolders have "[grouping]" or there's one subfolder only

[EndOfLine369]

I don't agree with 'one subfolder only' len(subdirs)>1 as that can cause many unwanted additions from current behavior for most current users. You'd make people like me have to create dummy folders to excluder single folder directories and not to mention those would create unmatched entries with random names like 'Extras'. A [grouping] or [multi-releases] in the grouping folder name should be good as that is a purpose driven choice to break from the expected scanner behavior.

[ZeroQI]

@EndOfLine369: i agree multiple folders depth equate generally to uncared data or dump folders. "Extras" folder are always ignored though. After looking at your reply, i find un-cleaned folders imbricated do gain to be ignored by default, and i find your insight valuable EndOfLine369, as always. So we have: if not match and len(reverse_path)>1 and path.count(os.sep) and not "[grouping]" in reverse_path[1] and not "[multi-releases]" in reverse_path[0]: #if grouping folders, skip.User to add them as additional folders

[ZeroQI]

Published to github scanner beta branch, please test

[dgw]

Is the beta branch pushed? GitHub says there is only one branch, master, updated 2 days ago.

[ZeroQI]

now it is. agent location changed to agent data folders / logs / series and sorted per series+guid Please test and feedback

[EndOfLine369]

Ok, found a way for Plex to scan in folders in a grouping folder as separate series. https://github.com/ZeroQI/Absolute-Series-Scanner/commit/597327ca3939c5c6caf93e2f0d8604d14835f9ba##commitcomment-23812316

Ran into this issue when I was trying to put together an emulator scanner. Basically, you have scan all files from the root call every time and block any separate folder call Plex tried to make.

This will also allow for the file lists from the scanner to always be up-to-date at every scan.

[EndOfLine369]

@ZeroQI, Pull Request https://github.com/ZeroQI/Absolute-Series-Scanner/pull/106

[EndOfLine369]

Also made change https://github.com/ZeroQI/Absolute-Series-Scanner/commit/0d0324f0bfc06c78f7da14ed8ed80392e6a4bc50

[EndOfLine369]

@dgw, Check out the current beta scanner. https://raw.githubusercontent.com/ZeroQI/Absolute-Series-Scanner/Beta/Scanners/Series/Absolute%20Series%20Scanner.py

To have a grouping folder: (1) the grouping folder must be in the library root folder (2) the grouping folder must have the string "[grouping]" in its name

EX: ('t1' was my testing root)

Edit: Think i finally got it working as expected. Had season folder issues.

[EndOfLine369]

@ZeroQI, as this manual scan from a grouping folder existance does a recursive scan, do you see any reason not to block a scan of any folders deeper than 3? EX: \<root>/<grouping[depth1]>/<series[depth2]>/<season[depth3]>/\<ignore>...

Committed in to reduce the scan time of useless directories. https://github.com/ZeroQI/Absolute-Series-Scanner/commit/ad9a8c605477ca47005109abf80487e910edd5f4

[EndOfLine369]

@ZeroQI , actually I have put a check right at the start of the scanner for any Scan call to stop any depth >3. https://github.com/ZeroQI/Absolute-Series-Scanner/commit/e2113b8a293a0b0e53c7091968f5aff143c54aa9 Should make it faster in general instead of trying to run through all the code to get where it eventually sees it as an undeclared grouping folder and then skips.

[EndOfLine369]

Additional commit on folder depths in general to save on speed. https://github.com/ZeroQI/Absolute-Series-Scanner/commit/7511ed2762a16aadb92f2a19e5319a6d317a0feb If in root grouping folder >3, else >2. EX: \<root>/<grouping[depth1]>/<series[depth2]>/<season[depth3]>/\<ignore>... \<root>/<series[depth1]>/<season[depth2]>/\<ignore>... Also put in 'is_grouping_scan' into Scan args so we don't have to test the root's subfolder names on each recursive run.

[EndOfLine369]

Has been committed for a while now. Any objections to deleting the legacy version you put in @ZeroQI? If no, will delete then close this issue ticket.

[ZeroQI]

Legacy supports .plexignore. Also supports grouping folders without any modification... Wantddvto keep it around for these reasons

On 17 Sep 2017 06:24, "EndOfLine369" notifications@github.com wrote:

Has been committed for a while now. Any objections to deleting the legacy version you put in @ZeroQI https://github.com/zeroqi? If no, will delete then close this issue ticket.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ZeroQI/Absolute-Series-Scanner/issues/103#issuecomment-330022238, or mute the thread https://github.com/notifications/unsubscribe-auth/AE7LZSs-3f3KijKP3UY5g6eykAZZ6Hlnks5sjKz6gaJpZM4O2N4y .

[EndOfLine369]

Then just closing