evilhero / mylar

An automated Comic Book downloader (cbr/cbz) for use with SABnzbd, NZBGet and torrents
GNU General Public License v3.0
975 stars 172 forks source link

'Enable Completed Download Handling' option doesnt't delete original cbr file and doesn't set permissions #1864

Closed verbumfeit closed 6 years ago

verbumfeit commented 6 years ago

I have activated the 'Enable Completed Download Handling' option, no category set for sabnzbd, the 'move' option and tagging enabled for post-processing. Renaming is not activated. The Post-processing seems to work fine, converts to .czb, tags it and moves it to my comic directory, but the .cbr file from the sabnzbd completed folder does not get removed and the .czb in the final directory has 600 permissions instead of the configured 664.

If I disable meta-tagging, everything works as expected. I'm on the latest development branch.

Here's the debug log:

2018-02-23 11:19:45 INFO [POST-PROCESSING] Post-Processing completed for: Aquaman issue: 11
2018-02-23 11:19:45 INFO Telegram notifications sent.
2018-02-23 11:19:45 DEBUG writing: {'Status': 'Downloaded', 'Location': u'Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz'} -- {'IssueID': u'558382'}
2018-02-23 11:19:45 INFO [POST-PROCESSING][UPDATER] Updating Status (Downloaded) now complete for Aquaman issue: 11
2018-02-23 11:19:45 INFO [POST-PROCESSING][UPDATER] Setting status to Downloaded in history.
2018-02-23 11:19:45 DEBUG [POST-PROCESSING][UPDATER] issueid: 558382
2018-02-23 11:19:45 DEBUG [POST-PROCESSING][UPDATER] comicid: 91738
2018-02-23 11:19:45 INFO incremented havefiles: 21
2018-02-23 11:19:45 INFO status: Snatched
2018-02-23 11:19:45 INFO totalfiles: 34
2018-02-23 11:19:45 DEBUG Successfully changed ownership and permissions [verbumfeit:comics] / [0775 / 0664]
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] Tidying up. Deleting temporary cache directory : /opt/mylar/cache/mylar_nmdsX5
2018-02-23 11:19:45 ERROR [POST-PROCESSING] /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire not empty. Skipping removal of directory - this will either be caught in further post-processing or it will have to be manually deleted.
2018-02-23 11:19:45 DEBUG odir: /opt/mylar/cache/mylar_nmdsX5 [self.nzb_folder: /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire]
2018-02-23 11:19:45 DEBUG File Option: move [META-ENABLED: True]
2018-02-23 11:19:45 DEBUG [FREESPACE-CHECK] /home/verbumfeit/Comics/DC Comics/Aquaman (2016) has 24.2GiB free
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] nfilename:Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] ofilename:Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] odir: /opt/mylar/cache/mylar_nmdsX5
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] self.nzb_folder: /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] Destination: /home/verbumfeit/Comics/DC Comics/Aquaman (2016)/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] Source: /opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz
2018-02-23 11:19:45 INFO [POST-PROCESSING][DIRECTORY-CHECK] Found comic directory: /home/verbumfeit/Comics/DC Comics/Aquaman (2016)
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] New Filename: Aquaman 011 (2017) (Digital) (BlackManta-Empire)
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] Rename Files is not enabled - keeping original filename.
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] Original Extension: .cbz
2018-02-23 11:19:45 DEBUG [POST-PROCESSING] Original Filename: Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz
2018-02-23 11:19:45 INFO [POST-PROCESSING] Sucessfully wrote metadata to .cbz (Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz) - Continuing..
2018-02-23 11:19:45 INFO [META-TAGGER][COMIC-TAGGER] Successfully wrote Comicbooklover tagging [/opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz]
2018-02-23 11:19:45 INFO None
2018-02-23 11:19:45 INFO /opt/mylar/lib/comictaggerlib/../bs4/init.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.The code that caused this warning is on line 5 of the file /opt/mylar/comictagger.py. To get rid of this warning, change code that looks like this:BeautifulSoup([your markup])to this:BeautifulSoup([your markup], "html.parser")markup_type=markup_type))
2018-02-23 11:19:41 DEBUG [META-TAGGER] Absolute path to script: /usr/bin/python
2018-02-23 11:19:41 DEBUG [META-TAGGER] Executing command: ['/usr/bin/python', '/opt/mylar/comictagger.py', '-s', '-m', 'volume=8', '--cv-api-key', u'REDACTED', '-o', '--id', u'558382', '--type', 'cbl', u'/opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz']
2018-02-23 11:19:41 DEBUG [META-TAGGER] Enabling ComicTagger script: ['/usr/bin/python', '/opt/mylar/comictagger.py'] with options: ['-s', '-m', 'volume=8', '--cv-api-key', u'REDACTED', '-o', '--id', u'558382', '--type', 'cbl', u'/opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz']
2018-02-23 11:19:41 INFO [META-TAGGER] Comicbooklover tagging meta-tagging processing started.
2018-02-23 11:19:41 INFO [META-TAGGER][COMIC-TAGGER] Successfully wrote ComicRack tagging [/opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz]
2018-02-23 11:19:41 INFO None
2018-02-23 11:19:41 INFO /opt/mylar/lib/comictaggerlib/../bs4/init.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("html.parser"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.The code that caused this warning is on line 5 of the file /opt/mylar/comictagger.py. To get rid of this warning, change code that looks like this:BeautifulSoup([your markup])to this:BeautifulSoup([your markup], "html.parser")markup_type=markup_type))
2018-02-23 11:19:37 DEBUG [META-TAGGER] Absolute path to script: /usr/bin/python
2018-02-23 11:19:37 DEBUG [META-TAGGER] Executing command: ['/usr/bin/python', '/opt/mylar/comictagger.py', '-s', '-m', 'volume=8', '--cv-api-key', u'REDACTED', '-o', '--id', u'558382', '--type', 'cr', u'/opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz']
2018-02-23 11:19:37 DEBUG [META-TAGGER] Enabling ComicTagger script: ['/usr/bin/python', '/opt/mylar/comictagger.py'] with options: ['-s', '-m', 'volume=8', '--cv-api-key', u'REDACTED', '-o', '--id', u'558382', '--type', 'cr', u'/opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz']
2018-02-23 11:19:37 INFO [META-TAGGER] ComicRack tagging meta-tagging processing started.
2018-02-23 11:19:37 DEBUG [META-TAGGER][COMIC-TAGGER][CBR-TO-CBZ] New filename: /opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz
2018-02-23 11:19:37 DEBUG [META-TAGGER][COMIC-TAGGER] : Archive exported successfully to: Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz (Original deleted)
2018-02-23 11:19:37 INFO None
2018-02-23 11:19:37 INFO Archive exported successfully to: Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbz (Original deleted)
2018-02-23 11:19:35 DEBUG [META-TAGGER] Absolute path to script: /usr/bin/python
2018-02-23 11:19:35 DEBUG [META-TAGGER] Executing command: ['/usr/bin/python', '/opt/mylar/comictagger.py', '-e', '--delete-rar', u'/opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbr']
2018-02-23 11:19:35 DEBUG [META-TAGGER] Enabling ComicTagger script: ['/usr/bin/python', '/opt/mylar/comictagger.py'] with options: ['-e', '--delete-rar', u'/opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbr']
2018-02-23 11:19:35 DEBUG [META-TAGGER] CBL Tagging enabled.
2018-02-23 11:19:35 DEBUG [META-TAGGER] CR Tagging enabled.
2018-02-23 11:19:35 DEBUG [META-TAGGER] ComicTagger 1.20.1 [ninjas.walk.alone / SHURIKEN] being used - using personal ComicVine API key supplied via mylar.
2018-02-23 11:19:35 DEBUG [META-TAGGER] Running the ComicTagger Add-on for Mylar
2018-02-23 11:19:35 DEBUG [META-TAGGER] comicpath : /opt/mylar/cache/mylar_nmdsX5
2018-02-23 11:19:35 DEBUG [META-TAGGER] sabnzbdscriptpath : /opt/mylar
2018-02-23 11:19:35 DEBUG [META-TAGGER] downloadpath : /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:35 DEBUG [META-TAGGER] scriptname : Mylar.py
2018-02-23 11:19:35 DEBUG [META-TAGGER] Paths / Locations:
2018-02-23 11:19:35 INFO New_Filepath: /opt/mylar/cache/mylar_nmdsX5/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbr
2018-02-23 11:19:35 INFO New_Folder: /opt/mylar/cache/mylar_nmdsX5
2018-02-23 11:19:35 INFO Filename: Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbr
2018-02-23 11:19:35 INFO Filepath: /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbr
2018-02-23 11:19:35 DEBUG [META-TAGGER] Filename is : /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbr
2018-02-23 11:19:35 DEBUG ComicTagger Path location for internal comictagger.py set to : /opt/mylar/comictagger.py
2018-02-23 11:19:35 DEBUG [META-TAGGER] dirName:/opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Metatagging enabled - proceeding...
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] ofilename: Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbr
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] odir: /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] ofilename: Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbr
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] odir (root): /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] New format: $Series $Issue ($Year)
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Not an annual - removing from filename parameters
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Comic Version: v8
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Comic Location: /home/verbumfeit/Comics/DC Comics/Aquaman (2016)
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Year: 2016
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Series: Aquaman
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Publisher: DC Comics
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Issue Year : 2017
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Pretty Comic Issue is : 11
2018-02-23 11:19:35 DEBUG Zero level supplement set to none.Issue will be set as : 11
2018-02-23 11:19:35 DEBUG issue detected greater than 10, but less than 100
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Zero Suppression set to : none
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] issuenumOG: 11
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] issueid: 558382
2018-02-23 11:19:35 INFO [POST-PROCESSING] [1/1] Starting Post-Processing for Aquaman issue: 11
2018-02-23 11:19:35 INFO [DUPECHECK] Duplication detection returned no hits. This is not a duplicate of anything that I have scanned in as of yet.
2018-02-23 11:19:35 INFO [DUPECHECK] Duplicate check for /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:35 INFO [POST-PROCESSING] issuenzb found.
2018-02-23 11:19:35 INFO [PPINFO-POST-PROCESSING-ATTEMPT] {'publisher': None, 'comicname': u'Aquaman', 'issueid': u'558382', 'comiclocation': None, 'sarc': None, 'issuenumber': u'11', 'oneoff': None, 'comicid': u'91738'}
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] Issueid: 558382
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] After conversions, nzbname is : Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:35 DEBUG [NZBNAME] nzbname (remove extensions, double spaces, convert underscores to spaces): Aquaman 011 2017 Digital BlackManta-Empire
2018-02-23 11:19:35 DEBUG [NZBNAME] nzbname (remove chars): Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:35 DEBUG [NZBNAME]: Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] nzb folder: /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:35 DEBUG [POST-PROCESSING] nzb name: Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:35 INFO File successfully downloaded - now initiating completed downloading handling.
2018-02-23 11:19:35 INFO location found @ /opt/sabnzbd/.sabnzbd/Downloads/complete/Aquaman.011.2017.Digital.BlackManta-Empire/Aquaman 011 (2017) (Digital) (BlackManta-Empire).cbr
2018-02-23 11:19:35 INFO found matching completed item in history. Job has a status of Completed
2018-02-23 11:19:35 INFO File has now downloaded!
-- -- --
2018-02-23 11:19:30 INFO eta: 11:19 Fr 23 Feb
2018-02-23 11:19:30 INFO timeleft: 0:00:00
2018-02-23 11:19:30 INFO mbleft: 0.00
2018-02-23 11:19:30 INFO status: Downloading
2018-02-23 11:19:30 INFO queue_params: {'output': 'json', 'search': u'SABnzbd_nzo_q1i88q', 'apikey': u'b327d84ea63c13a7d1a9b1c2acf1734c', 'mode': 'queue'}
2018-02-23 11:19:25 INFO eta: 11:19 Fr 23 Feb
2018-02-23 11:19:25 INFO timeleft: 0:00:08
2018-02-23 11:19:25 INFO mbleft: 34.14
2018-02-23 11:19:25 INFO status: Downloading
2018-02-23 11:19:25 INFO queue_params: {'output': 'json', 'search': u'SABnzbd_nzo_q1i88q', 'apikey': u'b327d84ea63c13a7d1a9b1c2acf1734c', 'mode': 'queue'}
2018-02-23 11:19:25 INFO Queue mbleft : 34.14
2018-02-23 11:19:25 INFO Queue status : Downloading
2018-02-23 11:19:25 INFO queue: {u'queue': {u'noofslots_total': 0, u'diskspace2_norm': u'24.3 G', u'paused': False, u'have_warnings': u'0', u'speedlimit_abs': u'', u'slots': [], u'speed': u'4.1 M', u'size': u'44.9 MB', u'rating_enable': False, u'refresh_rate': u'1', u'start': 0, u'version': u'2.3.2', u'diskspacetotal2': u'101.72', u'cache_max': u'1073741824', u'diskspacetotal1': u'101.72', u'status': u'Downloading', u'finish': 0, u'cache_art': u'17', u'sizeleft': u'34.1 MB', u'finishaction': None, u'paused_all': False, u'quota': u'0 ', u'have_quota': False, u'mbleft': u'34.14', u'diskspace2': u'24.27', u'diskspace1': u'24.27', u'scripts': [u'None', u'Deobfuscate.py', u'Sample-PostProc.py', u'autoProcessComics.py', u'ComicRN.py', u'autoProcessComics.cfg', u'autoProcessComics.pyc'], u'loadavg': u'0.78 | 0.84 | 0.71 | V=2581M R=90M', u'categories': [u'*', u'tv', u'movies', u'audio', u'software'], u'timeleft': u'0:00:08', u'pause_int': u'0', u'noofslots': 0, u'mb': u'44.86', u'eta': u'11:19 Fr 23 Feb', u'limit': 0, u'kbpersec': u'4233.22', u'speedlimit': u'100', u'cache_size': u'10.4 MB', u'left_quota': u'0 ', u'diskspace1_norm': u'24.3 G', u'queue_details': u'0'}}
2018-02-23 11:19:25 INFO successfully queried the queue for status
2018-02-23 11:19:20 INFO [UPDATER] Updated the status (Snatched) complete for Aquaman Issue: 11
2018-02-23 11:19:20 DEBUG [UPDATER] Provider is nzb.su
2018-02-23 11:19:20 INFO [UPDATER] Updating status to snatched
2018-02-23 11:19:20 DEBUG [UPDATER] issueid: 558382
2018-02-23 11:19:20 DEBUG [UPDATER] comicid: 91738
2018-02-23 11:19:20 INFO passing to updater.
2018-02-23 11:19:20 DEBUG findit = found!
2018-02-23 11:19:20 INFO setting the alternate nzbname for this download grabbed by nzb.su in the nzblog to : Aquaman0112017DigitalBlackManta-Empire
2018-02-23 11:19:20 INFO parameters set to {'status': True, 'queue': {'output': 'json', 'search': u'SABnzbd_nzo_q1i88q', 'apikey': u'b327d84ea63c13a7d1a9b1c2acf1734c', 'mode': 'queue'}, 'nzo_id': u'SABnzbd_nzo_q1i88q'}
2018-02-23 11:19:20 INFO setting the nzbid for this download grabbed by nzb.su in the nzblog to : bb7cf7eb078f28e020bf2d7cdb51ef43
2018-02-23 11:19:20 INFO sending now to http://localhost:8080/api
2018-02-23 11:19:20 DEBUG Found matching comic...preparing to send to Updater with IssueID: 558382 and nzbname: Aquaman.011.2017.Digital.BlackManta-Empire[Aquaman0112017DigitalBlackManta-Empire]
2018-02-23 11:19:20 INFO Now loading from queue: {'status': True, 'queue': {'output': 'json', 'search': u'SABnzbd_nzo_q1i88q', 'apikey': u'b327d84ea63c13a7d1a9b1c2acf1734c', 'mode': 'queue'}, 'nzo_id': u'SABnzbd_nzo_q1i88q'}
2018-02-23 11:19:20 INFO Successfully sent nzb file to SABnzbd
2018-02-23 11:19:20 INFO {u'status': True, u'nzo_ids': [u'SABnzbd_nzo_q1i88q']}
2018-02-23 11:19:20 DEBUG Recieved API command: downloadNZB
2018-02-23 11:19:20 DEBUG Matched to key. Api set to : download mode.
2018-02-23 11:19:20 INFO sending now to http://localhost:8080/api
2018-02-23 11:19:20 INFO parameters set to {'nzbname': u'Aquaman.011.2017.Digital.BlackManta-Empire.nzb', 'apikey': u'b327d84ea63c13a7d1a9b1c2acf1734c', 'name': u'http://192.168.178.55:8090/api?apikey=0bee33ad75c285c9c24900334e41903c&cmd=downloadNZB&nzbname=Aquaman.011.2017.Digital.BlackManta-Empire.nzb', 'cmd': 'downloadNZB', 'priority': '-100', 'mode': 'addurl', 'output': 'json'}
2018-02-23 11:19:20 DEBUG Successfully changed ownership and permissions [verbumfeit:comics] / [0775 / 0664]
2018-02-23 11:19:20 DEBUG Cache Directory successfully found at : /opt/mylar/cache. Ensuring proper permissions.
2018-02-23 11:19:20 INFO filen: Aquaman0112017DigitalBlackManta-Empire -- nzbname: Aquaman.011.2017.Digital.BlackManta-Empire are not identical. Storing extra value as : Aquaman0112017DigitalBlackManta-Empire
2018-02-23 11:19:20 DEBUG [FILENAME] end nzbname: Aquaman0112017DigitalBlackManta-Empire
2018-02-23 11:19:20 DEBUG [FILENAME] nzbname (\s): Aquaman0112017DigitalBlackManta-Empire
2018-02-23 11:19:20 DEBUG [FILENAME] filename (remove chars): Aquaman0112017DigitalBlackManta-Empire
2018-02-23 11:19:20 DEBUG Status code returned: 200
2018-02-23 11:19:19 INFO Download URL: https://api.nzb.su/getnzb/bb7cf7eb078f28e020bf2d7cdb51ef43.nzb&i=163792&r=xYOUDONTNEEDTOKNOWTHISx [VerifySSL:True]
2018-02-23 11:19:19 DEBUG link given by: nzb.su
2018-02-23 11:19:19 INFO Found Aquaman (2017) issue: 11 using nzb.su
2018-02-23 11:19:19 DEBUG issues match!
2018-02-23 11:19:19 DEBUG nzbname used for post-processing:Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:19 DEBUG [SEARCHER] end nzbname: Aquaman.011.2017.Digital.BlackManta-Empire
2018-02-23 11:19:19 DEBUG [SEARCHER] nzbname (\s): Aquaman 011 2017 Digital BlackManta-Empire
2018-02-23 11:19:19 DEBUG [SEARCHER] nzbname (remove chars): Aquaman 011 2017 Digital BlackManta-Empire
2018-02-23 11:19:19 DEBUG [SEARCHER] entry[title]: Aquaman 011 (2017) (Digital) (BlackManta-Empire)
2018-02-23 11:19:19 DEBUG integer value of issue we have found : 11000
2018-02-23 11:19:19 DEBUG issue we found for is : 011
2018-02-23 11:19:19 DEBUG integer value of issue we are looking for : 11000
2018-02-23 11:19:19 DEBUG issue we are looking for is : 11
2018-02-23 11:19:19 DEBUG this should be a match!
2018-02-23 11:19:19 DEBUG it's a go captain... - we matched 100.0%!
2018-02-23 11:19:19 DEBUG Match to series : 100.0 %.
2018-02-23 11:19:19 DEBUG splitit-len:1
2018-02-23 11:19:19 DEBUG scount:1
2018-02-23 11:19:19 DEBUG word matched on : aquaman
2018-02-23 11:19:19 DEBUG 0 Comparing: aquaman .to. aquaman
2018-02-23 11:19:19 DEBUG scount : 0
2018-02-23 11:19:19 DEBUG splitit: aquaman
2018-02-23 11:19:19 DEBUG Watchlist-length: 1
2018-02-23 11:19:19 DEBUG search-length: 1
2018-02-23 11:19:19 DEBUG length match..proceeding
2018-02-23 11:19:19 DEBUG len-watchcomic : 1
2018-02-23 11:19:19 DEBUG splitst : 1
2018-02-23 11:19:19 DEBUG ['aquaman'] watchlist word count: 1
2018-02-23 11:19:19 DEBUG ['aquaman', '011'] nzb series word count: 1
2018-02-23 11:19:19 DEBUG adjusting from: aquaman 011 to: 011
2018-02-23 11:19:19 DEBUG adjusted nzb comic and issue: aquaman 011
2018-02-23 11:19:19 DEBUG original nzb comic and issue: aquaman 011
2018-02-23 11:19:19 DEBUG Scanner detected: blackmanta-empire
2018-02-23 11:19:19 DEBUG 5. Bracket Word: blackmanta-empire
2018-02-23 11:19:19 DEBUG digital edition detected
2018-02-23 11:19:19 DEBUG 3. Bracket Word: digital
2018-02-23 11:19:19 DEBUG 2017 - right years match baby!
2018-02-23 11:19:19 DEBUG year looking for: 2017
2018-02-23 11:19:19 DEBUG year detected: 2017
2018-02-23 11:19:19 DEBUG 1. Bracket Word: 2017
2018-02-23 11:19:19 DEBUG ComVersChk : 8
2018-02-23 11:19:19 DEBUG UseFuzzy is : None
2018-02-23 11:19:19 DEBUG Comic: aquaman 011
2018-02-23 11:19:19 DEBUG 0. Bracket Word: aquaman 011
2018-02-23 11:19:19 DEBUG Cleantitle: aquaman 011 (2017) (digital) (blackmanta-empire)
2018-02-23 11:19:19 DEBUG Cleantitle: aquaman 011 (2017) (digital) (blackmanta-empire)
2018-02-23 11:19:19 DEBUG Entry: Aquaman 011 (2017) (Digital) (BlackManta-Empire)
2018-02-23 11:19:19 DEBUG Wed, 16 Nov 2016 18:26:25 -0500 is after store date of 2016-11-16
2018-02-23 11:19:19 DEBUG size given as: 44.9 MB
2018-02-23 11:19:19 INFO comsize_b: 47035664
2018-02-23 11:19:19 DEBUG sub:Aquaman 011 (2017) (Digital) (BlackManta-Empire)
2018-02-23 11:19:19 DEBUG checking search result: Aquaman 011 (2017) (Digital) (BlackManta-Empire)
2018-02-23 11:19:19 DEBUG ComVersChk : 8
2018-02-23 11:19:19 DEBUG UseFuzzy is : None
2018-02-23 11:19:19 DEBUG Comic: comic aquaman and the others 011
2018-02-23 11:19:19 DEBUG 0. Bracket Word: comic aquaman and the others 011
2018-02-23 11:19:19 DEBUG Cleantitle: comic aquaman and the others 011
2018-02-23 11:19:19 DEBUG detected invalid nzb filename - attempting to detect year to continue
2018-02-23 11:19:19 DEBUG Cleantitle: comic aquaman and the others 011
2018-02-23 11:19:19 DEBUG Removed extension from filename: .cbr
-- -- --
2018-02-23 11:19:19 DEBUG Entry: [COMIC] Aquaman and the Others 011.cbr
2018-02-23 11:19:19 DEBUG Sat, 10 Dec 2016 12:19:47 -0500 is after store date of 2016-11-16
2018-02-23 11:19:19 DEBUG size given as: 48.6 MB
2018-02-23 11:19:19 INFO comsize_b: 50992128
2018-02-23 11:19:19 DEBUG sub:[COMIC] Aquaman and the Others 011.cbr
2018-02-23 11:19:19 DEBUG checking search result: [COMIC] Aquaman and the Others 011.cbr
2018-02-23 11:19:19 INFO no errors on data retrieval...proceeding
2018-02-23 11:19:19 DEBUG [SSL: True] Search URL: https://api.nzb.su/api?t=search&q=aquaman%20011&o=xml&cat=7030&apikey=xUDONTNEEDTOKNOWTHISx
2018-02-23 11:19:19 INFO pausing for 60 seconds before continuing to avoid hammering
2018-02-23 11:19:19 DEBUG findloop: 0 / findcount: 1
2018-02-23 11:19:19 INFO Shhh be very quiet...I'm looking for Aquaman issue: 11 (2017) using nzb.su
2018-02-23 11:19:19 DEBUG tmp_prov_count: 1
2018-02-23 11:19:19 DEBUG prov_count: 0
2018-02-23 11:19:19 DEBUG cmloopit: 2
2018-02-23 11:19:19 DEBUG Initiating Search via : api
2018-02-23 11:19:19 DEBUG srchloop: 2
2018-02-23 11:19:19 DEBUG calpha/cnumber: None / None
2018-02-23 11:19:19 INFO search provider order is ['nzb.su']
2018-02-23 11:19:19 DEBUG Usenet Retention : 1500 days
2018-02-23 11:19:19 DEBUG there are : 1 nzb providers you have selected.
2018-02-23 11:19:19 DEBUG there are : 0 torrent providers you have selected.
2018-02-23 11:19:19 DEBUG nzbprovider(s): ['nzb.su']
2018-02-23 11:19:19 DEBUG Checking for torrent enabled.
2018-02-23 11:19:19 INFO Issue Title given as : the deluge prelude: condition critical
2018-02-23 11:19:19 DEBUG Publisher is : DC Comics
2018-02-23 11:19:19 INFO Initiating manual search for Aquaman issue: 11
2018-02-23 11:19:19 DEBUG mode:want
2018-02-23 11:19:19 DEBUG ComicID:91738
verbumfeit commented 6 years ago

Cleanup now works, thank you!

The permission issue persists though, let me know if I can provide additional information to help.

evilhero commented 6 years ago

If it only occurs during metatagging, then I would think that it could be a perms issue. Since mylar copies the file from the original location to the cache, it might be not be inheriting perms properly, or the user running mylar might not have enough perms to do one of the actions. I believe mylar takes the existing perms and uses this on creation (when copying from the original location to the cache during metatagging)

What are the perms/owmership of the cache mylar_xxxxx folder and file(s) inside?

verbumfeit commented 6 years ago

The mylar_xxxxx folder always has 700 perms and verbumfeit:comics ownership, which is the user:group mylar is running under.

The file inside has 666 perms at first, owned by sabnzbd:comics, which is the user:group sabnzbd is running under. After tagging (I presume) the perms are set to 600, with verbumfeit:comics ownership.

Both users verbumfeit and sabnzbd are members of the comics-group.

evilhero commented 6 years ago

Because it initially copies the file from the original sab location to the cache, it would also copy the existing ownership/permission (which in this case would be sabnzbd:comics / 666). It then performs metatagging by creating the tmp dir with 700 perms (this is due to the mkdtemp command that Mylar utilizes). It then unzips the file, and recreates a new file - with the identical permissions (0600) I would imagine since it's a straight file creation. Then it moves the file to the destination, and tries to chown/chmod it. So if I'm following this right, the user running mylar (verbumfeit) would not have the proper authority to perform a chown since it's owner-based, and not a group-based command.

Again, I'm not 100% sure exactly what's going on - it would seem to imply something with the perms is stopping it from chmodding/chowning the file properly after the initial metatag. All of this is probably in large part due to the restrictive nature of the temp file usage that Mylar uses. Since only the user who created the tmp dir can alter the ownership (not the group, just the owner) - it might be restricting it that way too. I might be able to force a chmod on the file while it's in the tmp directory, and then when it gets moved to the series folder it could be applied properly.....

(sorry for rambling, was trying to work thru this in my head while typing and think I got more confused than not)

EDIT. Actually can you verify the permissions on the file that's moved to the series directory (the one it's trying to chmod/chown and not doing). Since that would be the problematic file, it'd be good to verify that that's correct. The owner stays the same, so it should still be able to perform a chown/chmod you would think.

verbumfeit commented 6 years ago

The file in the series directory has 600 and verbumfeit:comics permissions. So it should have no problems executing the chmod.

I discovered that mylar calls the setperms(path, dir) function during post-processing only once with dir=False argument (which should be correct, since we just downloaded and created a file). Then os.walk(path) is called with path being /comicdirectory/DC Comics/Aquaman/Aquaman 011.cbz, so the variables for dirs and files stay empty and the chmod/chwon never gets executed. See here: https://github.com/evilhero/mylar/blob/cd5bbfe74b807586a07707aea7886d32f8401885/mylar/filechecker.py#L1395-L1403

What would work is replacing the code above with this for the folder (If we really need it here, since it's only a file we want to chmod/chown):

permission = int(mylar.CONFIG.CHMOD_DIR, 8)
os.chown(os.path.dirname(path), chowner, chgroup)
os.chmod(os.path.dirname(path), permission)

And this for the file:

permission = int(mylar.CONFIG.CHMOD_FILE, 8)
os.chown(path, chowner, chgroup)
os.chmod(path, permission)

I don't know if this would break something else, especially since we have the same block of code again a few lines down: https://github.com/evilhero/mylar/blob/cd5bbfe74b807586a07707aea7886d32f8401885/mylar/filechecker.py#L1395-L1414

EDIT: I wrote a fix that won't break existing functionality. Have to check if it works properly when I get home, then I will submit it :) EDIT2: Here we go: #1870

verbumfeit commented 6 years ago

Fixed with e1e2916f07a15545ebacbfa94886eec8b87a9e40