Closed LeeThompson closed 5 years ago
evilhero
commented
6 years ago Subfolder post-processing works fine, so odds are it's localized.
How are you post-processing - using completed download handling or comicRN?
Also debug logs help greatly, and are beneficial in troubleshooting. Just listing the filename and/or the error itself without the context leading up to it is equivalent to bringing your car to a mechanic and saying you hear a rattling sound.
LeeThompson
commented
6 years ago I have debug log on and I will be updating this with debug logs below.
My settings:
(If I've set anything stupid please let me know)
Notes
Some of the issue seems to be in ComicTagger which I am not familiar with at all.
Extra File Handling If there's extra stuff downloaded (.nzb, .par files, etc) those files and the folder aren't removed. If anything is unexpected when ComicTagger extracts the archive for processing (convert to cbz etc), it aborts but leaves the file. Shouldn't it delete and mark as failed?
Unexpected Extension One comic downloaded as a .rar and it just sat there, should rename to ,cbr. (I renamed it to .cbr and it automatically processed it shortly thereafter). (The best solution would be for an option to have Mylar identify the archive type [zip or rar], verify integrity (test), and then rename the extension to the correct eComic extension. If it's not an archive or intact, mark as failed and delete.)
Logs
In this sample, a comic is downloaded but comictagger doesn't like it. I did test the archive with winrar and it tested clean so I really don't know what it was unhappy about.
In any event, if it thinks it's a bad copy, there should be an option so it that Mylar marks as failed and it should delete it.
23-May-2018 08:23:12 - INFO :: mylar.duplicate_filecheck.2088 : Thread-29 : [DUPECHECK] Duplicate check for /downloads/mylar/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr
23-May-2018 08:23:12 - INFO :: mylar.duplicate_filecheck.2215 : Thread-29 : [DUPECHECK] Duplication detection returned no hits. This is not a duplicate of anything that I have scanned in as of yet.
23-May-2018 08:23:12 - INFO :: mylar.Process_next.1639 : Thread-29 : [POST-PROCESSING] [1/7] Starting Post-Processing for Marvel Graphic Novel issue: 17
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1642 : Thread-29 : [POST-PROCESSING] issueid: 117738
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1643 : Thread-29 : [POST-PROCESSING] issuenumOG: 17
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1723 : Thread-29 : [POST-PROCESSING] Zero Suppression set to : none
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1763 : Thread-29 : issue detected greater than 10, but less than 100
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1778 : Thread-29 : Zero level supplement set to none.Issue will be set as : 17
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1795 : Thread-29 : [POST-PROCESSING] Pretty Comic Issue is : 17
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1798 : Thread-29 : [POST-PROCESSING] Issue Year : 1985
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1805 : Thread-29 : [POST-PROCESSING] Publisher: Marvel
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1809 : Thread-29 : [POST-PROCESSING] Series: Marvel Graphic Novel
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1817 : Thread-29 : [POST-PROCESSING] Year: 1982
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1820 : Thread-29 : [POST-PROCESSING] Comic Location: /comics/new/Marvel Graphic Novel (1982)
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1823 : Thread-29 : [POST-PROCESSING] Comic Version: None
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1832 : Thread-29 : [POST-PROCESSING] No version # found for series, removing from filename
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1833 : Thread-29 : [POST-PROCESSING] New format is now: $Series $Annual $Issue ($Year)
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1841 : Thread-29 : [POST-PROCESSING] Not an annual - removing from filename parameters
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1842 : Thread-29 : [POST-PROCESSING] New format: $Series $Issue ($Year)
23-May-2018 08:23:12 - DEBUG :: mylar.Process_next.1909 : Thread-29 : [POST-PROCESSING] Metatagging enabled - proceeding...
23-May-2018 08:23:12 - DEBUG :: mylar.run.26 : Thread-29 : [META-TAGGER] dirName:/downloads/mylar
23-May-2018 08:23:12 - DEBUG :: mylar.run.31 : Thread-29 : ComicTagger Path location for internal comictagger.py set to : /app/mylar/comictagger.py
23-May-2018 08:23:12 - DEBUG :: mylar.run.35 : Thread-29 : [META-TAGGER] Filename is : /downloads/mylar/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr
23-May-2018 08:23:12 - DEBUG :: mylar.run.50 : Thread-29 : Filepath: /downloads/mylar/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr
23-May-2018 08:23:12 - DEBUG :: mylar.run.51 : Thread-29 : Filename: Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr
23-May-2018 08:23:12 - DEBUG :: mylar.run.53 : Thread-29 : New_Folder: /config/mylar/cache/mylar_gcFpZi
23-May-2018 08:23:12 - DEBUG :: mylar.run.55 : Thread-29 : New_Filepath: /config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr
23-May-2018 08:23:12 - DEBUG :: mylar.run.73 : Thread-29 : [META-TAGGER] Paths / Locations:
23-May-2018 08:23:12 - DEBUG :: mylar.run.74 : Thread-29 : [META-TAGGER] scriptname : Mylar.py
23-May-2018 08:23:12 - DEBUG :: mylar.run.75 : Thread-29 : [META-TAGGER] downloadpath : /downloads/mylar
23-May-2018 08:23:12 - DEBUG :: mylar.run.76 : Thread-29 : [META-TAGGER] sabnzbdscriptpath : /app/mylar
23-May-2018 08:23:12 - DEBUG :: mylar.run.77 : Thread-29 : [META-TAGGER] comicpath : /config/mylar/cache/mylar_gcFpZi
23-May-2018 08:23:12 - DEBUG :: mylar.run.78 : Thread-29 : [META-TAGGER] Running the ComicTagger Add-on for Mylar
23-May-2018 08:23:12 - DEBUG :: mylar.run.119 : Thread-29 : [META-TAGGER] ComicTagger 1.20.1 [ninjas.walk.alone / SHURIKEN] being used - using personal ComicVine API key supplied via mylar.
23-May-2018 08:23:12 - DEBUG :: mylar.run.136 : Thread-29 : [META-TAGGER] CBL Tagging enabled.
23-May-2018 08:23:12 - DEBUG :: mylar.run.190 : Thread-29 : [META-TAGGER] Enabling ComicTagger script: ['/usr/bin/python', '/app/mylar/comictagger.py'] with options: ['-e', '--delete-rar', u'/config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr']
23-May-2018 08:23:12 - DEBUG :: mylar.run.198 : Thread-29 : [META-TAGGER] Executing command: ['/usr/bin/python', '/app/mylar/comictagger.py', '-e', '--delete-rar', u'/config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr']
23-May-2018 08:23:12 - DEBUG :: mylar.run.199 : Thread-29 : [META-TAGGER] Absolute path to script: /usr/bin/python
23-May-2018 08:23:13 - DEBUG :: mylar.run.237 : Thread-29 : [META-TAGGER] Output: readArchiveFile(): [file is not expected size: 531436 vs 0] /config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr:marvel heroes - revenge of the living monolith gn/revenge of the living monolith gn fc.jpg [attempt # 1]
readArchiveFile(): [file is not expected size: 531436 vs 0] /config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr:marvel heroes - revenge of the living monolith gn/revenge of the living monolith gn fc.jpg [attempt # 2]
readArchiveFile(): [file is not expected size: 531436 vs 0] /config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr:marvel heroes - revenge of the living monolith gn/revenge of the living monolith gn fc.jpg [attempt # 3]
readArchiveFile(): [file is not expected size: 531436 vs 0] /config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr:marvel heroes - revenge of the living monolith gn/revenge of the living monolith gn fc.jpg [attempt # 4]
readArchiveFile(): [file is not expected size: 531436 vs 0] /config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr:marvel heroes - revenge of the living monolith gn/revenge of the living monolith gn fc.jpg [attempt # 5]
readArchiveFile(): [file is not expected size: 531436 vs 0] /config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr:marvel heroes - revenge of the living monolith gn/revenge of the living monolith gn fc.jpg [attempt # 6]
readArchiveFile(): [file is not expected size: 531436 vs 0] /config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbr:marvel heroes - revenge of the living monolith gn/revenge of the living monolith gn fc.jpg [attempt # 7]
Error while copying to /config/mylar/cache/mylar_gcFpZi/Marvel Graphic Novel 17 - Revenge of The Living Monolith (1985).cbz:
Archive failed to export!
23-May-2018 08:23:13 - ERROR :: mylar.Process_next.1937 : Thread-29 : [POST-PROCESSING] This is a corrupt archive - whether CRC errors or it is incomplete. Marking as BAD, and not post-processing.
Kinda long winded - but here goes (ala bullet form):
Extra file handling is not Mylar's job to cleanup. It looks for anything that's a cbr/cbz/webp and will attempt to post-process that and ignore anything else as that should up to the download client to clean up as required. It does this for several reasons, one of which being not everyone has just comic files in a given directory that they're post-processing and issuing a cleanup phase on a directory with other things is probably not a great idea. When it's successful, it will clean up just the file that it post-processed for the exact reason mentioned above.
Mylar will only initiate a failed download / retry when it's an issue that's being downloaded on. The reason is simple, if you download something outside of Mylar and it fails it has no idea on where to find the relevant information as there's no information pertaining to it in the db.
Mylar will abort the process if it's a corrupt archive, but it will perform cleanup if it has the proper permissions and the correct settings are enabled (ie. if you you're using comictagger). If you're running in a dockerized environment you have to ensure that the user running Mylar has complete access to the given directories affected so that it can actually remove items.
No, it should not rename .rar's to .cbr for the exact reason I mentioned above. Lots of users have their files mixed in with other stuff and Mylar just arbitrarily renaming .rar to .cbr just because it's a .rar would be considered back practice and dangerous. It's also for this reason that Mylar will only target cbr/cbz/webp when even querying indexers for results.
ComicTagger already does the archive type detection (if it's a cbz, but named as a cbr it will extract, and convert it to the proper extension), which is why it's used. Mylar used to do it itself, but it proved to be too problematic of a solution and there were more problems than solutions, so the CT method was implemented as it was already working.
The non-delete for corrupt issues was implemented at one point, so that users could confirm that it was infact corrupt (as some didn't believe it, or questioned why), as well as having a sample to draw from when troubleshooting began as to why it was considered corrupt. However that being said, it should be deleting upon retrieving a failed response from the comictagger addon as that was what it had been changed to and as far as I was aware - it was still working. Again - permissions will play a big part in being able to clean up items.
The file is not expected size error just started occurring over the last week or so. I've tried to troubleshoot it based on a file that I was provided with that exhibited this behaviour but couldn't figure out the exact reason as to why it was failing. My best guess is that it was compressed in a newer version of rar that the version of unrar that Mylar comes with (it's a python wrapper that basically just calls your OS unrar) is having problems listing the contents of the rar properly and thereby can' t retrieve the file size. Again this isn't Mylar that's doing this, this is ComicTagger's code. I am looking at this still though, and trying different things - so hopefully can figure out why it's failing on these cbr's and fix things.
The CT that comes with Mylar has been modified in order to accommodate Mylar usage and as such has specific calls and a small bit of code change in order to make things work. This was when CT hadn't been updated in awhile, so modifying was the only option. They now have a new fork of CT, which has a different code - but attempting to integrate it into Mylar is abit too much work atm as it requires a crapload of extra dependencies that I wouldn't be able to include as part of the Mylar package and even just trying to get things to work together is proving very problematic.
LeeThompson
commented
6 years ago I guess that makes sense, better to be too cautious with data than too liberal. (My own code I usually have that stuff default to cautious but have switches for situations where people need it.)
Some of Mylar's logs (and I know some of it is due to other software included) are a little confusing to me as to what it's unhappy about.
There may also be an issue with dupe handling but I need to do some more tests.
So I guess close this issue since there appear to be a number of other factors kicking around.
stoofz
commented
5 years ago I'm having a similar issue. Downloads go to /mnt/dl/nzb/comics. NZBGet is extracting the files into sub folders. Mylar is having an issue moving them from the sub folder and throwing the following errors. If I take the file out of the folder Mylar is able to post process. I'm not using ComicRN, and have post-processing (move) enabled. I am running the development branch.
| 2019-01-31 16:34:15 | ERROR | Uncaught exception: Traceback (most recent call last): File "/opt/Mylar/mylar/logger.py", line 337, in new_run old_run(*args, *kwargs) File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(self.args, **self.kwargs) File "/opt/Mylar/mylar/PostProcessor.py", line 1515, in Process if dupthis['action'] == 'dupe_src' or dupthis['action'] == 'dupe_file':TypeError: 'NoneType' object has no attribute 'getitem' |
|---|---|---|
| 2019-01-31 16:34:15 | WARNING | [DUPECHECK] Make sure if you are using ComicRN, you do not have Completed Download Handling enabled (or vice-versa). Aborting |
| 2019-01-31 16:34:15 | WARNING | [DUPECHECK] File cannot be located in location specified. Something has moved or altered the name. |
| 2019-01-31 16:34:15 | INFO | [DUPECHECK] Duplicate check for /Darth Vader 022 2018 Digital Kileko-Empire/Darth Vader 022 (2018) (Digital) (Kileko-Empire).cbr |
| 2019-01-31 16:34:15 | DEBUG | [POST-PROCESSING] There are 1 files found that match on your watchlist, 0 files are considered one-off's, and 0 files do not match anything |
| 2019-01-31 16:34:15 | DEBUG | [POST-PROCESSING][SUCCESSFUL MATCH: Darth Vader-101960] Match verified for Darth Vader 022 (2018) (Digital) (Kileko-Empire).cbr |
| 2019-01-31 16:34:15 | DEBUG | [POST-PROCESSING][DEFINITIVE-NAME MATCH] Definitive name match exactly to : Darth Vader [101960] |
| 2019-01-31 16:34:15 | DEBUG | [POST-PROCESSING][SUB: /Darth Vader 022 2018 Digital Kileko-Empire][CLOCATION: /mnt/dl/nzb/comics] |
barbequesauce
commented
5 years ago @stoofz please open a new issue and link it to this one... and make sure to follow the directions in the issue template...
I have Mylar working with NZBGet, it is set so downloads go to /download/mylar
Unfortunately NZBGet is insisting on putting each comic in it's own folder, for example:
/download/mylar/Legion.of.Super-Heroes.Annual.01.2011.Minutemen-CalamityCoyote/Legion of Super-Heroes Annual 01 (2011) (Minutemen-CalamityCoyote).cbrThis seems to greatly confuse Mylar for some reason.