Beta Scanner - Issues (Scanner Hang)

sven-7 commented 6 years ago

Not really sure how to log this, but here is what I experienced:

Using latest Beta HAMA as well.

Slow scanning/hang. Scanner would hang immediately after starting. It looks like in the Plex scan logs, it hands when it finds a "doesn't exist, can't skip" despite that it is there and does exist. A Plex dance on the show that it says might resolve the issue, it might not. If I switch to the master branch, this issue does not exist. Switching the scanner seems to make this issue come up.

For example. it will hit:

Feb 19, 2018 09:01:39.472 [9504] DEBUG - Adding subdirectory for scanner: "E:\Series\Green Green"
Feb 19, 2018 09:01:39.472 [9504] DEBUG - Adding subdirectory for scanner: "E:\Series\Grimgar of Fantasy and Ash"
Feb 19, 2018 09:01:39.472 [9504] DEBUG - Adding subdirectory for scanner: "E:\Series\Guilty Crown"
Feb 19, 2018 09:01:39.472 [9504] DEBUG - Adding subdirectory for scanner: "E:\Series\Gun Frontier"
Feb 19, 2018 09:01:39.472 [9504] DEBUG - Adding subdirectory for scanner: "E:\Series\Gundam Build Fighters"
Feb 19, 2018 09:01:39.472 [9504] DEBUG - Adding subdirectory for scanner: "E:\Series\Gundam Reconguista in G"
Feb 19, 2018 09:01:39.472 [9504] DEBUG - Adding subdirectory for scanner: "E:\Series\Gundam-san"
Feb 19, 2018 09:01:39.472 [9504] DEBUG - Adding subdirectory for scanner: "E:\Series\Gungrave"
Feb 19, 2018 09:01:39.472 [9504] DEBUG - Adding subdirectory for scanner: "E:\Series\Gunparade March"
Feb 19, 2018 09:01:39.472 [9504] DEBUG - Adding subdirectory for scanner: "E:\Series\Gurren Lagan"
Feb 19, 2018 09:01:39.473 [9504] DEBUG - Adding file for scanner: "E:\Series\X-Plex-Token.id"
Feb 19, 2018 09:01:39.476 [9504] DEBUG - File 'E:\Series\Gangsta\Gangsta. - 01 - Naughty Boys.mkv' didn't exist, can't skip.
Feb 19, 2018 09:03:29.677 [9504] DEBUG -       * Scanning 07-Ghost Season 1 Episode 1
Feb 19, 2018 09:03:29.677 [9504] DEBUG - Looking for path match for [E:\Series\07-Ghost\07-Ghost - 1 - The Future of Painful Thoughts is.mkv]
Feb 19, 2018 09:03:29.679 [9504] DEBUG - Path matched, we're reusing media item 164350
Feb 19, 2018 09:03:29.688 [9504] DEBUG -       * Scanning 07-Ghost Season 1 Episode 10

You can see all activity stops for about two minutes. If I Plex dance Gangtsa, then another show takes it's place. All of the files it suggests do not exist, are present.

Update Note: while it is hung, it does appear during that time the individual logs are generating, but it still looks like nothing is happening. The scan wheel on Plex is just spinning. It seems like Plex will scan a bit, then generate logs, then start again. A total scan takes around 10 minutes at times.

logs.zip

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/55226664-beta-scanner-issues-scanner-hang?utm_campaign=plugin&utm_content=tracker%2F3327432&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F3327432&utm_medium=issues&utm_source=github).

mikethecalamity commented 6 years ago

Do you have the latest Scanner? Mine was super slow last week before I did that.

sven-7 commented 6 years ago

Yeah - I grabbed the latest beta as of this posting to try it out. It's still acting this way now.

ZeroQI commented 6 years ago

Cannot see in logs, if master branch scanner works will do a new scanner using more proper function calls

sven-7 commented 6 years ago

Are there any logs I can provide to give you better clarity?

ZeroQI commented 6 years ago

The very method used is problematic. The issue is if there is an error, it removes all files from the library and it needs to be recreated even if the scanner error is fixed. Also the error is not always indicated making it a royal pain in the ass to develop...

Nearly finished though, i will keep you posted Current progress is there: https://gist.github.com/ZeroQI/974f6335ad4f31ecdfe1e4af0404098d

sven-7 commented 6 years ago

Great, thanks! I can test whenever it's ready.

ZeroQI commented 6 years ago

Updated beta scanner [https://gist.github.com/ZeroQI/974f6335ad4f31ecdfe1e4af0404098d] and is working for normal series and plexignore support (and useless zip support) Grouping folders do not work yet and random folders in season folders should be working but not without season folders. Please test while i try to add grouping folder support.

sven-7 commented 6 years ago

It's still hanging for me. It continues to get stuck on a missing file that actually isn't missing.

This is from Logs\Plex Media Scanner.log

Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Granblue Fantasy The Animation"
Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gravitation TV"
Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Great Teacher Onizuka"
Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Green Green"
Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Grimgar of Fantasy and Ash"
Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Guilty Crown"
Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gun Frontier"
Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gundam Build Fighters"
Feb 27, 2018 13:03:36.265 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gundam Reconguista in G"
Feb 27, 2018 13:03:36.265 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gundam-san"
Feb 27, 2018 13:03:36.265 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gungrave"
Feb 27, 2018 13:03:36.265 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gunparade March"
Feb 27, 2018 13:03:36.265 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gurren Lagann"
Feb 27, 2018 13:03:36.269 [12956] DEBUG - File 'F:\Series\Haikyuu!!\Haikyuu!! - 1 - The End & the Beginning.mkv' didn't exist, can't skip.

ZeroQI commented 6 years ago

Could be corrupted Plex database believe the ep is missing un plex database

On 27 Feb 2018 18:04, "sven-7" notifications@github.com wrote:

It's still hanging for me. It continues to get stuck on a missing file that actually isn't missing.

This is from Logs\Plex Media Scanner.log

Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Granblue Fantasy The Animation" Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gravitation TV" Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Great Teacher Onizuka" Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Green Green" Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Grimgar of Fantasy and Ash" Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Guilty Crown" Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gun Frontier" Feb 27, 2018 13:03:36.264 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gundam Build Fighters" Feb 27, 2018 13:03:36.265 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gundam Reconguista in G" Feb 27, 2018 13:03:36.265 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gundam-san" Feb 27, 2018 13:03:36.265 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gungrave" Feb 27, 2018 13:03:36.265 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gunparade March" Feb 27, 2018 13:03:36.265 [12956] DEBUG - Adding subdirectory for scanner: "E:\Series\Gurren Lagann" Feb 27, 2018 13:03:36.269 [12956] DEBUG - File 'F:\Series\Haikyuu!!\Haikyuu!! - 1 - The End & the Beginning.mkv' didn't exist, can't skip.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ZeroQI/Absolute-Series-Scanner/issues/133#issuecomment-368970596, or mute the thread https://github.com/notifications/unsubscribe-auth/AE7LZTetJR4HYseBAPUROKrgj-LOCR8sks5tZEO9gaJpZM4SKmid .

sven-7 commented 6 years ago

I checked for corruptions, per the Plex instructions, but everything came back okay.

Also, now every show with a .id file is being added as an S0E501

ZeroQI commented 6 years ago

if only leaving one root folder on f:, does it work or still complain? Corrected the above and posted correction on gist (same link)

sven-7 commented 6 years ago

As in only F:/Series/ ?

I have four drives with a Series folder on them and then all series sub-folders listed inside.

ZeroQI commented 6 years ago

yes, as it could be a bug in my code when it attack the 2nd folder or there is something in the filenames on f giving issues

sven-7 commented 6 years ago

Ah. So right now my library is set to scan four harddrives. Each drive has one main folder called Series. That is the only folder on each drive.

See below:

sven-7 commented 6 years ago

It definitely does look like it gets caught on one drive before moving onto the next.

ZeroQI commented 6 years ago

so only f gives an error? would need the filelist then...

sven-7 commented 6 years ago

No, they all give errors.

ZeroQI commented 6 years ago

Please test https://gist.github.com/ZeroQI/974f6335ad4f31ecdfe1e4af0404098d grouping folder support while keeping caching if ep folders present in season folder Plexignore seem to work Should be better system wise

sven-7 commented 6 years ago

I am still having all .id files added as S0E501's.

Edit - Regarding the drives, it seems to list one error per drive. It changes if I Plex Dance the show. On the old scanner, the error exists, but it doesn't stall things.

sven-7 commented 6 years ago

I made a two series library across two different root drives. It supplies the scan error during it, but only after a second harddrive was added.

After the Rain's .id file also shows up as a special.

Logs.zip

sven-7 commented 6 years ago

Here's a theory -- watching it live, it seems to scan one drive then create and write all the individual .filelist and .scanner log files. Then it does that three more times over the course of all my drives. That might explain the pause between drives.

The .id file showing up as a special only exists on the current version of the beta scanner. It does not exist on the prior d23868d commit.

ZeroQI commented 6 years ago

just published on beta branch one without .plexignore and id file issue grouping folders write twice the filelist. trying to solve that

ZeroQI commented 6 years ago

Published a new version, logging all scan calls in both filelist and scanner logs, much simpler to follow now and if it crashed we would have no filelist if only gathering for grouped folders. Should not detect any non video ext as file now, small issue with splitext ignoring leading dots Please let me know any issues

sven-7 commented 6 years ago

Not seeing any lingering issues on my end.

The scanner still seems to do things one HDD at a time. The process seems to be:

Scan E:/Series
Make .scanner and .filelist for all in E:/Series
Slight Pause
Scan F:/Series etc...

No .id files on the latest scanner. So good to go there.

ZeroQI commented 6 years ago

worked all day, added .plexignore full support. Please let me know any issues

ZeroQI commented 6 years ago

Zip file support and logging fixed, and full .plexignore support

This should be stable. Worked both day fully... Reverted to scan all while on root scanner call by recalling only relevant folders as i had either series no grouping folders or the other way around...

https://github.com/ZeroQI/Absolute-Series-Scanner/blob/Beta/Scanners/Series/Absolute%20Series%20Scanner.py

Please test and feedback if faster (<2s per series with couples very long ones), it really took time to make this one...

sven-7 commented 6 years ago

It definitely seems a lot faster. It look a few times to get through things. Here is what happened:

On the first scan, it removed about 300 series that I had. The overall scan was several times faster in creating the logs. If I were to guess, it removed everything after my first drive (out of four).
On the second scan, it added back about 150 of them. Some of them only partially.
On the third scan, it got back up to the original number of series, with the exception of one episode I added during the first scan.
On the fourth scan, it picked up the episode and everything looks to be back to normal.
Fifth scan was fairly quick. I think it's gone from about 10-12 minutes down to 3ish minutes. I have 490 series right now, for perspective on how much is being scanned.

Edit: Starting to do a lot of Plex Dancing to clear out some of the bad posters that got downloaded. Just completed the first drive and everything went superbly with the scanner.

ZeroQI commented 6 years ago

@sven-7 Thanks for letting me know. Glad it is faster. Since it is scanned manually from the root folder i believe it had to un-do then re-do which it didn't like. Would be nice to create a separate library to scan from the start with that version.

Looking now to make it faster with simple optimisations like cancelling the Plex naturally called scan instances, possibly not helping there, so we do things once and only if needed...

sven-7 commented 6 years ago

No problem!

One more note on the Plex Dance. I removed an entire drive's worth of series. The scan deleted them all out, as expected. Then it ran once I re-added them and went through the scan and agent process. When I did the next drive, it removed all the series from drive 1, which it just completed. After the drive 2 Plex dance process was completed and I hit re-scan, the Drive 1 series were added again. The same thing happened with Drive 2 when I went to do the third. Not sure why it is deleting and re-adding twice.

ZeroQI commented 6 years ago

That would explain similar issue i have been having on a single root folder library By plex dance do we mean the same?

delete library
Clean bundle
empty trash
optimise database
recreate the library adding all root folders at once I get the feeling the behaviour of Plex in regards to the scanner changed... Never wittnessed similar... Optimizing further but not at home so will post tomorrow the update after testing

sven-7 commented 6 years ago

I am not deleting the library (I didn't want to affect my watch counts). I'm following the standard Plex Dance per the forums.

It does seem like Plex has changed something with the scanner. It's odd that it needs to add-delete-then add again. Other than that, your scanner is working great.

ZeroQI commented 6 years ago

@sven-7 makes sense now. You can however create a second library that point to the same folders for tests i believe. So from what i understand it is better apart for issues while migrating an already existing library? Any functionnality i could implement from another third party scanners or even plex ones ?

sven-7 commented 6 years ago

Yes - I will likely create a smaller subset library to test on.

Correct - migrating an existing library may result in what seems like a double-scan. Thus far, it does it once, and then it's good to go.

sven-7 commented 6 years ago

Getting a weird error and super slow scanner all of a sudden...

From _root_scanner.log:

Library: 'Anime Series', root: 'E:\Series', path: '', files: '0', dirs: '181', Plex scan date: 2018-03-03 14:32:04
=============================================================================================================================================================
[no files detected]
-------------------------------------------------------------------------------------------------------------------------------------------------------------
07-Ghost                                                    , subdir_files:  25, reverse_path: ['07-Ghost']                            
91 Days                                                     , subdir_files:  13, reverse_path: ['91 Days']                             
A Certain Magical Index                                     , subdir_files:  48, reverse_path: ['A Certain Magical Index']             
A Certain Scientific Railgun                                , subdir_files:  48, reverse_path: ['A Certain Scientific Railgun']        
A Dark Rabbit Has Seven Lives                               , subdir_files:  12, reverse_path: ['A Dark Rabbit Has Seven Lives']       
A place further than the Universe                           , subdir_files:   9, reverse_path: ['A place further than the Universe']   
Abenobashi Mahou Shoutengai                                 , subdir_files:  13, reverse_path: ['Abenobashi Mahou Shoutengai']         
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
ACCA 13 -Territory Inspection Department                    , subdir_files:  12, reverse_path: ['ACCA 13 -Territory Inspection Department']
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Accel World                                                 , subdir_files:  24, reverse_path: ['Accel World']                         
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Ace of Diamond                                              , subdir_files: 126, reverse_path: ['Ace of Diamond']                      
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Active Raid                                                 , subdir_files:  24, reverse_path: ['Active Raid']                         
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Afro Samurai                                                , subdir_files:   5, reverse_path: ['Afro Samurai']                        
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
After the Rain                                              , subdir_files:   8, reverse_path: ['After the Rain']                      
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Ai Tenchi Muyo!                                             , subdir_files:  60, reverse_path: ['Ai Tenchi Muyo!']                     
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Air Gear                                                    , subdir_files:  25, reverse_path: ['Air Gear']                            
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Ajin                                                        , subdir_files:  26, reverse_path: ['Ajin']                                
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Akashic Records of Bastard Magic Instructor                 , subdir_files:  12, reverse_path: ['Akashic Records of Bastard Magic Instructor']
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Alderamin on the Sky                                        , subdir_files:  13, reverse_path: ['Alderamin on the Sky']                
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Aldnoah Zero                                                , subdir_files:  24, reverse_path: ['Aldnoah Zero']                        
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Alice & Zouroku                                             , subdir_files:  12, reverse_path: ['Alice & Zouroku']                     
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
All Out!!                                                   , subdir_files:  25, reverse_path: ['All Out!!']                           
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Amagi Brilliant Park                                        , subdir_files:  13, reverse_path: ['Amagi Brilliant Park']                
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
And you thought there is never a girl online                , subdir_files:  12, reverse_path: ['And you thought there is never a girl online']
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Angel Beats!                                                , subdir_files:  16, reverse_path: ['Angel Beats!']                        
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
anohana The Flower We Saw That Day                          , subdir_files:  11, reverse_path: ['anohana The Flower We Saw That Day']  
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Another                                                     , subdir_files:  12, reverse_path: ['Another']                             
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Arata The Legend                                            , subdir_files:  12, reverse_path: ['Arata The Legend']                    
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Arc The Lad                                                 , subdir_files:  26, reverse_path: ['Arc The Lad']                         
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Area 88 (2004)                                              , subdir_files:  12, reverse_path: ['Area 88 (2004)']                      
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Argento Soma                                                , subdir_files:  26, reverse_path: ['Argento Soma']                        
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Aria - The Natural                                          , subdir_files:  26, reverse_path: ['Aria - The Natural']                  
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Aria the Scarlet Ammo                                       , subdir_files:  12, reverse_path: ['Aria the Scarlet Ammo']               
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Armed Librarians Book of Bantorra                           , subdir_files:  27, reverse_path: ['Armed Librarians Book of Bantorra']   
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Assassination Classroom                                     , subdir_files:  47, reverse_path: ['Assassination Classroom']             
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Astro Fighter Sunred                                        , subdir_files:  52, reverse_path: ['Astro Fighter Sunred']                
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Asura Cryin                                                 , subdir_files:  26, reverse_path: ['Asura Cryin']                         
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Attack on Titan                                             , subdir_files:  42, reverse_path: ['Attack on Titan']                     
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Avatar The Last Airbender                                   , subdir_files:  60, reverse_path: ['Avatar The Last Airbender']           
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
B The Beginning                                             , subdir_files:  12, reverse_path: ['B The Beginning']                     
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Baby Steps                                                  , subdir_files:  50, reverse_path: ['Baby Steps']                          
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Baccano!                                                    , subdir_files:  16, reverse_path: ['Baccano!']                            
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series
Baki the Grappler                                           , subdir_files:  49, reverse_path: ['Baki the Grappler']                   
CACHE_PATH is not a valid folder: C:\Users\RAFTEL3\AppData\Local\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems\_Logs\Anime Series

sven-7 commented 6 years ago

I added the logs back, but it's still slow.

ZeroQI commented 6 years ago

https://github.com/ZeroQI/Absolute-Series-Scanner/blob/Beta/Scanners/Series/Absolute%20Series%20Scanner.py

Please try this version and report any ongoing issue.

ZeroQI commented 6 years ago

Don't seem to have the problem... just uploaded new version. Please let me know any issue. still scan in less than 2s per series including bleach

sven-7 commented 6 years ago

Will do. The one earlier was still seeing issues where the item gets scanned in, then disappearing. It seems like items that get scanned in on partial scans disappear at some point. I have to manually scan or wait until the next scheduled for it to come back.

Will test the latest.

sven-7 commented 6 years ago

Still having items removed on their first scan after being added.

See Plex removing it here:

Mar 12, 2018 12:48:04.644 [7548] DEBUG - Path matched, we're reusing media item 252564
Mar 12, 2018 12:48:04.649 [7548] DEBUG - Removing 0 media items that were left.
Mar 12, 2018 12:48:04.649 [7548] DEBUG - Removing 1 directories that were left.
Mar 12, 2018 12:48:04.649 [7548] DEBUG - Deleting media item 269185.
Mar 12, 2018 12:48:04.650 [7548] DEBUG - Was connected to metadata item 284444, count is now 0.
Mar 12, 2018 12:48:04.650 [7548] DEBUG - Destroying metadata item 284444 (Cause and Effect)
Mar 12, 2018 12:48:04.651 [7548] DEBUG - Updating deletion state for metadata item 275879, is has a dead item count of 0.
Mar 12, 2018 12:48:04.651 [7548] DEBUG - Updating deletion state for metadata item 275878, is has a dead item count of 0.
Mar 12, 2018 12:48:04.652 [9608] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 12, 2018 12:48:04.652 [7548] DEBUG - Deleting directory [GARO -VANISHING LINE-] (6757)
Mar 12, 2018 12:48:04.653 [9608] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 12, 2018 12:48:04.675 [9608] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/notify/destroy?librarySectionID=2&ids=284444&types=4&parentIDs=275879&rootIDs=275878
Mar 12, 2018 12:48:04.676 [7548] DEBUG - Refreshing section 2 of type: 2
Mar 12, 2018 12:48:04.676 [9608] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/:/metadata/notify/destroy?librarySectionID=2&ids=284444&types=4&parentIDs=275879&rootIDs=275878

I've added the logs in and labeled them pre and post.

Logs.zip

ZeroQI commented 6 years ago

could you create a separate test library maybe pointing to the same folder ? I do not have these changing behavior at all...

sven-7 commented 6 years ago

It is the same with a test library.

Added 20 episodes of Garo. Added the 21st. Then the 21st dropped off. After another scan it came back.

ZeroQI commented 6 years ago

@sven-7 thanks a lot for all the tests. Super weird result though... i could scan all while on root folder scan but nothing is cached and will take a long time for every scan, i am unsure what to do... What is the best version of the scanner speed and stability wise? the current version supports plexignore and can be adapted easily... I believe i need to choose between grouping folders and fast/cacheable scanner calls...

EndOfLine369 commented 6 years ago

@ZeroQI or both and go back to how I originally added in grouping folder support. Where it defaults to standard Plex driven fast/cacheable scanner calls unless a trigger (string or file) is found for it to run as a grouping folder root full scan.

sven-7 commented 6 years ago

I think this was the version the add/delete/add started in: db068a6 or ff0e71c. The speed was fixed in db068a6 and subsequent improvements.

Edit: Same thing again:

Add episode of Boruto to library.
Plex initiates partial library scan to add Boruto. HAMA pulls down metadata. Log: Boruto Add - Scan 1.log.zip
Self initiated scan.
Upon scan completion, episode of Boruto disappears.

Mar 14, 2018 13:33:42.108 [10404] DEBUG - Deleting media item 269255.
Mar 14, 2018 13:33:42.108 [10404] DEBUG - Was connected to metadata item 284547, count is now 0.
Mar 14, 2018 13:33:42.108 [10404] DEBUG - Destroying metadata item 284547 (Wasabi and Namida)
Mar 14, 2018 13:33:42.109 [10404] DEBUG - Updating deletion state for metadata item 272138, is has a dead item count of 0.
Mar 14, 2018 13:33:42.110 [10404] DEBUG - Updating deletion state for metadata item 272102, is has a dead item count of 0.
Mar 14, 2018 13:33:42.110 [8660] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 14, 2018 13:33:42.110 [10404] DEBUG - Deleting directory [Boruto Naruto Next Generations] (6768)
Mar 14, 2018 13:33:42.111 [8660] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 14, 2018 13:33:42.134 [8660] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/notify/destroy?librarySectionID=2&ids=284547&types=4&parentIDs=272138&rootIDs=272102
Mar 14, 2018 13:33:42.135 [10404] DEBUG - Refreshing section 2 of type: 2
Mar 14, 2018 13:33:42.135 [8660] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/:/metadata/notify/destroy?librarySectionID=2&ids=284547&types=4&parentIDs=272138&rootIDs=272102
Mar 14, 2018 13:33:42.384 [5912] DEBUG - Refreshing 0 IDs.

Self initiate second scan.
Boruto episode is added back to libary.

Mar 14, 2018 13:39:44.837 [5012] DEBUG -       * Scanning Boruto Naruto Next Generations [tvdb4-321285] Season 4 Episode 48
Mar 14, 2018 13:39:44.837 [5012] DEBUG - Looking for path match for [E:\Series\Boruto Naruto Next Generations\Boruto Naruto Next Generations - 48 - The Genin Documentary!.mkv]
Mar 14, 2018 13:39:44.837 [5012] DEBUG - Path matched, we're reusing media item 269039
Mar 14, 2018 13:39:44.841 [5012] DEBUG -       * Scanning Boruto Naruto Next Generations [tvdb4-321285] Season 4 Episode 49
Mar 14, 2018 13:39:44.841 [5012] DEBUG - Looking for path match for [E:\Series\Boruto Naruto Next Generations\Boruto Naruto Next Generations - 49 - Wasabi and Namida.mkv]
Mar 14, 2018 13:39:44.841 [5012] DEBUG - Skipping hash check, no size match for 573998753 bytes.
Mar 14, 2018 13:39:44.841 [5012] DEBUG - No match for hash.
Mar 14, 2018 13:39:44.842 [5012] DEBUG - Creating show 'Boruto Naruto Next Generations [tvdb4-321285]'
Mar 14, 2018 13:39:44.842 [5012] DEBUG - Downloading document http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.842 [5012] DEBUG - HTTP requesting GET http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.843 [5012] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.845 [5012] DEBUG - Added new metadata item (Boruto Naruto Next Generations [tvdb4-321285]) with ID 284548
Mar 14, 2018 13:39:44.846 [5012] DEBUG - Downloading document http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.846 [5012] DEBUG - HTTP requesting GET http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.846 [1384] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 14, 2018 13:39:44.847 [1384] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 14, 2018 13:39:44.847 [5012] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.847 [5012] DEBUG - Updating metadata item (save) (Boruto Naruto Next Generations [tvdb4-321285]) with ID 284548
Mar 14, 2018 13:39:44.848 [1384] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 14, 2018 13:39:44.848 [5012] DEBUG - Creating season 4 for show 'Boruto Naruto Next Generations [tvdb4-321285]'
Mar 14, 2018 13:39:44.848 [5012] DEBUG - Downloading document http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.848 [5012] DEBUG - HTTP requesting GET http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.849 [1384] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 14, 2018 13:39:44.859 [5012] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.860 [5012] DEBUG - Added new metadata item () with ID 284549
Mar 14, 2018 13:39:44.861 [1384] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 14, 2018 13:39:44.861 [5012] DEBUG - Downloading document http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.861 [5012] DEBUG - HTTP requesting GET http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.862 [1384] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 14, 2018 13:39:44.862 [5012] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/library/changestamp
Mar 14, 2018 13:39:44.863 [5012] DEBUG - Added new metadata item (Wasabi And Namida) with ID 284550
Mar 14, 2018 13:39:44.863 [1384] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache
Mar 14, 2018 13:39:44.864 [1384] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/:/metadata/flushMetadataCounterCache

ZeroQI commented 6 years ago

@sven-7 Thanks for the tests, i trust you but it is difficult to resolve something i don't see... i spotted a difference between that and the version prior and call to scan was modified now though:

files[] is filename only
path is relatibe
dirs is list of fullpath dir

I hope it solves the issue, the files were added fullpath, could have messed things up a bit...

Also working on TheTVDB Api v2 (json) and better collection support

sven-7 commented 6 years ago

So...new problem, maybe this will help solve it?

Plex seems to initiate a scan, but nothing happens. I see it in the logs for Plex, but ASS doesn't do anything nor generate any logs. If I manually scan, it will then add.

Mar 15, 2018 19:05:14.989 [4344] INFO - Plex Media Scanner v1.12.1.4871-6daee94e2 - Microsoft PC x64 - build: windows-i386 - GMT -04:00
Mar 15, 2018 19:05:14.990 [4344] INFO - Windows version: 6.2 (Build 9200), language en-US
Mar 15, 2018 19:05:14.990 [4344] INFO - 8 4200 MHz processor(s): Architecture=0, Level=6, Revision=40457 Processor Identifier=Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
Mar 15, 2018 19:05:14.993 [4344] DEBUG - "C:\Program Files (x86)\Plex\Plex Media Server\Plex Media Scanner.exe" --scan --refresh --section 2 --directory "F:\Series\Laid-Back Camp" 
Mar 15, 2018 19:05:15.164 [12404] DEBUG - Opening 20 database sessions to library (com.plexapp.plugins.library), SQLite 3.13.0, threadsafe=1
Mar 15, 2018 19:05:15.242 [12404] DEBUG - Scanning Anime Series using en(Absolute Series Scanner) with 0 current media items and 1 section locations in the database...
Mar 15, 2018 19:05:15.242 [11640] DEBUG - HTTP requesting GET http://127.0.0.1:32400/:/metadata/notify/cullTimeline?librarySectionID=2&sinceTime=1521153915
Mar 15, 2018 19:05:15.244 [11640] DEBUG - HTTP 200 response from GET http://127.0.0.1:32400/:/metadata/notify/cullTimeline?librarySectionID=2&sinceTime=1521153915
Mar 15, 2018 19:05:15.978 [12404] DEBUG - Performing a scan with 'Absolute Series Scanner' (language: en virtual: 0).
Mar 15, 2018 19:05:15.978 [12404] DEBUG -   * Scanning F:\Series\Laid-Back Camp
Mar 15, 2018 19:05:15.982 [12404] DEBUG -     * Scanning directory F:\Series\Laid-Back Camp (parent: yes)
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 1 - Mount Fuji and Curry Noodles.mkv"
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 10 - Clumsy Travellers and Camp Meetings.mkv"
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 11 - Christmas Camp!.mkv"
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 2 - Welcome to the Outdoor Activities Club!.mkv"
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 3 - Mount Fuji and Relaxed Hot Pot Camp.mkv"
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 4 - The Outdoor Activities Club and the Solo Camping Girl.mkv"
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 5 - Two Camps, Two Campers' Views.mkv"
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 6 - Meat and Fall Colors and the Mystery Lake.mkv"
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 7 - A Night on the Lake Shore and Campers.mkv"
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 8 - Exams, Caribou, Steamed Buns, Yum!.mkv"
Mar 15, 2018 19:05:15.983 [12404] DEBUG - Adding file for scanner: "F:\Series\Laid-Back Camp\Laid-Back Camp - 9 - A Night of Navigator Nadeshiko and Hot Spring Steam.mkv"
Mar 15, 2018 19:05:15.985 [12404] DEBUG - Directory had 11 files, database had 0 files, can't skip.
Mar 15, 2018 19:05:15.988 [12404] DEBUG - Removing 0 media items that were left.
Mar 15, 2018 19:05:15.988 [12404] DEBUG - Removing 0 directories that were left.
Mar 15, 2018 19:05:15.988 [12404] DEBUG - Updating directory 'Laid-Back Camp' (ID 6774) to time 2018-03-15 19:04:14.
Mar 15, 2018 19:05:16.017 [12404] DEBUG - Refreshing section 2 of type: 2
Mar 15, 2018 19:05:16.280 [6132] DEBUG - Refreshing 0 IDs.

ZeroQI commented 6 years ago

I have installed Plex Media Server for windows since that is what you were using

i had an issue with scudlee movie collection, but resolved itself, but no issue on 2 separate test libraries... I don't want to mess up your plex database and stuff, but i cannot reproduce the issue and i need to know if i need to re-code the scanner or not...

in the meantime trying to code:

tvdb api v2 which uses json, might be able to load multiple language titles from thetvdb
trying to find a better systems for collections. Would love to use grouping folders as collections, BUT i need the scanner on point without using tags to flag grouping folders, so might need to further rely on your ongoing tests... but if scanner works i can group in plex series grouped on the hard drive, that would be briliant when you set the library to hide items in collections...
a way to deduce the library root from agent side without Plex token, would allow logs and to find grouping folders for collections...
Wrote a bit from traileradicts but no reply for an api key...

sven-7 commented 6 years ago

Tested the new one. Episodes still are not adding when added to the library. Plex is leaving logs, because I have "Run a partial scan when changes are detected" checked. ASS still isn't doing anything.

Also, everything in my library was deleted and re-added, which went smoothly and quickly. Just an FYI.

ZeroQI commented 6 years ago

@sven-7 Thanks a lot for your post, i realised i had removed from dirs all folders and that's why normal plex calls weren't made anymore to series folders and their subfolders so i was doing all manually... I thought it was because of the manual call wrongly...

Now the scanner call the folders only for grouping folders, and prevent normal scanner calls for these very folders. Only folders with actionable files are called so it is pretty optimised and take 4s instead of 25s now...

Please re-test but hopefully this is the one.

PS: if using .plexignore plex give scanner errors but it's not in my code and all works. Lost hours because i couldn't fix an error not caused by my code :/

sven-7 commented 6 years ago

It's working!!

I'm not seeing any lingering issues. Nothing is disappearing or not scanning anymore. I'm not seeing any errors. Speed seems to be good as well. Glad you were able to spot it. Sorry for my poor descriptions on the issue.

I think this is the one.

ZeroQI / Absolute-Series-Scanner

Beta Scanner - Issues (Scanner Hang) #133