search

evilhero / mylar

An automated Comic Book downloader (cbr/cbz) for use with SABnzbd, NZBGet and torrents
GNU General Public License v3.0
975 stars 172 forks source link

"Before post-processing" scripts can no longer change the downloaded filename #1202

Closed toddos closed 3 years ago

toddos commented 8 years ago

I prefer to store my comics in CBZ format rather than CBR. To that end, I have a before post-processing script that identifies if the download was a CBR file, unrars, zips, and renames the file to .cbz. This was previously working, but now I get this error during post processing:

Original Filename: some comic.cbr

Original Extension: .cbr

New Filename: Some Comic 001

Source:/mnt/public/downloads/comic-related/some comic.cbr

Destination:/mnt/nas/ebooks/Comics/Some Comic (2015)/Some Comic 001.cbr

Failed to move directory - check directories and manually re-run.

Post-Processing ABORTED.

Since my script changed "some comic.cbr" to "some comic.cbz", post-processing aborts.

I've attempted to write a post-post-processing script that will convert the file after the fact, but post-post-processing always fails with an error that the output is unparseable 


29-Jan-2016 21:16:08 - ERROR   :: Post-Processing : Uncaught exception: Traceback (most recent call last):
  File "/home/servers/mylar/mylar/mylar/logger.py", line 158, in new_run
    old_run(*args, **kwargs)
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/servers/mylar/mylar/mylar/PostProcessor.py", line 947, in Process
    return self.Process_next(comicid, issueid, issuenumOG)
  File "/home/servers/mylar/mylar/mylar/PostProcessor.py", line 1603, in Process_next
    self._run_extra_scripts(nzbn, self.nzb_folder, filen, folderp, seriesmetadata)
  File "/home/servers/mylar/mylar/mylar/PostProcessor.py", line 159, in _run_extra_scripts
    logger.fdebug(u"Script result: " + out)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 31: ordinal not in range(128)

This despite the fact that the output of the post script is nearly identical to the output of the pre script that doesn't fail parsing (output is unrar and 7za spew, for debugging purposes).

Is this no longer supported functionality?

evilhero commented 8 years ago

It's supported - just that with the last few commits and the changing of how comictagger interacts with Mylar,and Mylar changing the way it handles cbrtocbz conversions, some of the relevant backend code had to go through some fairly significant changes.

I'll have to look into the pre/post-script functionality again to ensure that things line up properly once more. Would you be able to post the debug log from the start of a post-processing run up until the error occurs? I'd need to see what it's trying to do, and if you already have a script that was working, I can go off of that to make sure it's working again...

toddos commented 8 years ago

From mylar.log (comic series/name has been sanitized, as this doesn't appear to be specific to any single comic)


29-Jan-2016 18:19:02 - INFO    :: Post-Processing : [POST-PROCESSING] issuenzb found.
29-Jan-2016 18:19:02 - INFO    :: Post-Processing : [DUPECHECK] Duplicate check for /mnt/public/downloads/comic-related/sanitized
29-Jan-2016 18:19:02 - INFO    :: Post-Processing : [DUPECHECK] Duplication detection returned no hits. This is not a duplicate of anything that I have scanned in as of yet.
29-Jan-2016 18:19:02 - INFO    :: Post-Processing : [POST-PROCESSING] [1/1] Starting Post-Processing for sanitized issue: 1
29-Jan-2016 18:19:09 - INFO    :: Post-Processing : [POST-PROCESSING][DIRECTORY-CHECK] Found comic directory: /mnt/nas/ebooks/Comics/sanitized (2015)
29-Jan-2016 18:19:09 - WARNING :: Post-Processing : [POST-PROCESSING] Failed to move directory : /mnt/public/downloads/comic-related/sanitized/sanitized.cbr to /mnt/nas/ebooks/Comics/sanitized (2015)/sanitized 001.cbr - check directory and manually re-run
29-Jan-2016 18:19:09 - WARNING :: Post-Processing : [POST-PROCESSING] Post-Processing ABORTED

And from the post-processing log in sab:


Loading config from /var/db/sabnzbd/script/autoProcessComics.cfg
Opening URL: http://servers:8090/post_process?nzb_name=sanitized&failed=0&apc_version=1.0&comicrn_version=1.01&nzb_folder=%2Fmnt%2Fpublic%2Fdownloads%2Fcomic-related%2Fsanitized
nzb name: sanitized

nzb folder: /mnt/public/downloads/comic-related/sanitized

nzbname: sanitized

issue detected less than 10

Zero level supplement set to 00x. Issue will be set as : 001

Issue Year: 2016

Publisher: sanitized

Series: sanitized

Year: 2015

Comic Location: /mnt/nas/ebooks/Comics/sanitized (2015)

Comic Version: None

No version # found for series - tag will not be available for renaming.

initiating pre script detection.

mylar.PRE_SCRIPTS : /home/servers/mylar/scripts/pre_cbrtocbz.sh

cmd to be executed: ['/bin/bash', '/home/servers/mylar/scripts/pre_cbrtocbz.sh', 'sanitized', '/mnt/public/downloads/comic-related/sanitized', "{'seriesmeta': [{'publisher': u'sanitized', 'issueid': u'000000', 'name': 'sanitized', 'issueyear': u'2016', 'comicyear': u'2015', 'issue': u'1', 'comicid': u'00000'}]}"]

Script result: /mnt/public/downloads/comic-related/sanitized ~/mylar/mylar

UNRAR 5.21 freeware      Copyright (c) 1993-2015 Alexander Roshal

Extracting from sanitized.cbr

Creating    /mnt/public/downloads/comic-related/sanitized/unrar/sanitized  OK

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-000.jpg       0%  1%  2%  3%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-001.jpg       3%  4%  5%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-002.jpg       5%  6%  7%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-003.jpg       7%  8%  9%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-004.jpg       9% 10% 11% 12% 13% 14% 15%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-005.jpg      15% 16%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-006.jpg      16% 17% 18%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-007.jpg      18% 19% 20% 21% 22%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-008.jpg      22% 23% 24% 25%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-009.jpg      25% 26% 27% 28%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-010.jpg      28% 29% 30% 31% 32% 33% 34% 35% 36%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-011.jpg      36% 37% 38% 39%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-012.jpg      39% 40% 41%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-013.jpg      41% 42% 43% 44%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-014.jpg      44% 45% 46% 47%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-015.jpg      47% 48% 49% 50%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-016.jpg      50% 51% 52%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-017.jpg      52% 53% 54% 55% 56% 57% 58% 59% 60%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-018.jpg      60% 61% 62%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-019.jpg      62% 63% 64%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-020.jpg      65% 66% 67%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-021.jpg      67% 68% 69% 70% 71%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-022.jpg      71% 72% 73%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-023.jpg      74% 75% 76% 77%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-024.jpg      77% 78% 79% 80%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-025.jpg      80% 81% 82% 83% 84%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-026.jpg      84% 85% 86% 87%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-027.jpg      87% 88% 89%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-028.jpg      89% 90% 91% 92% 93% 94% 95%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/sanitized 001-029.jpg      95% 96% 97%  OK 

Extracting  /mnt/public/downloads/comic-related/sanitized/unrar/sanitized/z.jpg      97% 98% 99%  OK 

All OK

/mnt/public/downloads/comic-related/sanitized/unrar /mnt/public/downloads/comic-related/sanitized ~/mylar/mylar

7-Zip (A) [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18

p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,1 CPU)

Scanning

Creating archive ../sanitized.cbz

Compressing  sanitized/sanitized 001-000.jpg

Compressing  sanitized/sanitized 001-001.jpg

Compressing  sanitized/sanitized 001-002.jpg

Compressing  sanitized/sanitized 001-003.jpg

Compressing  sanitized/sanitized 001-004.jpg

Compressing  sanitized/sanitized 001-005.jpg

Compressing  sanitized/sanitized 001-006.jpg

Compressing  sanitized/sanitized 001-007.jpg

Compressing  sanitized/sanitized 001-008.jpg

Compressing  sanitized/sanitized 001-009.jpg

Compressing  sanitized/sanitized 001-010.jpg

Compressing  sanitized/sanitized 001-011.jpg

Compressing  sanitized/sanitized 001-012.jpg

Compressing  sanitized/sanitized 001-013.jpg

Compressing  sanitized/sanitized 001-014.jpg

Compressing  sanitized/sanitized 001-015.jpg

Compressing  sanitized/sanitized 001-016.jpg

Compressing  sanitized/sanitized 001-017.jpg

Compressing  sanitized/sanitized 001-018.jpg

Compressing  sanitized/sanitized 001-019.jpg

Compressing  sanitized/sanitized 001-020.jpg

Compressing  sanitized/sanitized 001-021.jpg

Compressing  sanitized/sanitized 001-022.jpg

Compressing  sanitized/sanitized 001-023.jpg

Compressing  sanitized/sanitized 001-024.jpg

Compressing  sanitized/sanitized 001-025.jpg

Compressing  sanitized/sanitized 001-026.jpg

Compressing  sanitized/sanitized 001-027.jpg

Compressing  sanitized/sanitized 001-028.jpg

Compressing  sanitized/sanitized 001-029.jpg

Compressing  sanitized/z.jpg

Everything is Ok

/mnt/public/downloads/comic-related/sanitized ~/mylar/mylar

~/mylar/mylar

Original Filename: sanitized.cbr

Original Extension: .cbr

New Filename: sanitized

Source:/mnt/public/downloads/comic-related/sanitized/sanitized.cbr

Destination:/mnt/nas/ebooks/Comics/sanitized/sanitized 001.cbr

Failed to move directory - check directories and manually re-run.

Post-Processing ABORTED.

The script I'm using looks like this:


#!/bin/bash
folder=$2
cbrfiles=*.cbr

pushd "$folder"

shopt -s nullglob

for f in *.cbr
do
    base=$(basename "$f")
    cbzfile="../${base%.*}.cbz"
    mkdir "$folder/unrar"
    unrar x -y "$f" "$folder/unrar"
    pushd "$folder/unrar"
    7za a -tzip "$cbzfile" *
    popd
    rm -rf "$folder/unrar"
    mv "$f" "$f.converted"
done
popd

It's brute-force, but it worked previously because Mylar would look for any file with a known comic extension (well, cbr or cbz, not sure if it worked with cb7 or others) within the nzb directory rather than looking for a specific file. So my script takes "nzb file.cbr", unpacks it, zips it back to "nzb file.cbz", and then renames "nzb file.cbr" to "nzb file.cbr.converted" (just in case I needed the original for some reason).

As for the parsing error in post-post-processing, that output was identical to the output above starting from "Script result:", so I wouldn't have expected it to fail parsing where the pre script didn't. Honestly, though, I'd rather not use a post-post-processing script because mylar scans for files before the post-post script runs and thus for completeness I'd need to find a way to trigger Mylar to rescan for files in that book directory so that it sees the new cbz rather than the old cbr. The pre-script is my preference, or better yet built-in functionality to convert to a preferred format.

evilhero commented 8 years ago

Yes I did have to move some of the post-processing workflow around abit in order to accomodate the recent comictagger changes and how Mylar handles it. Looks like the pre-scripts got moved to after the 'check for .cbr/.cbz' files, so when it gets to the pre-script it already knows the file name and doesn't recheck.

What about if I get Mylar to check the output of the call command - and have your script return the full path (full directory path + filename + ext) of the new file? Mylar can then use that going forward in the post-processing and it won't hiccup again.

Of course, there are other alternatives as well. Mylar already does a cbr-to-cbz conversion using ComicTagger now (which is included with Mylar). So as long as you have configparser installed on your system, you don't have to do much else aside from enable the meta-tagging aspect and select the type of tagging (ComicRack xml or ComicBookLover cbl, or both). Then under the 'Post-Processing' of the same tab, select 'copy' in the 'When post-processing files' portion. This will copy the file out of the existing directory, move it into a temporary location, convert it there, then move said file from the temporary location into your series directory, and update the file count/checks.

Here's another thought: if you don't want Mylar to metatag, but you want it to convert cbr-to-cbz (or verify it's a valid cbz if that's what's being checked), I can always add an option under meta-tagging to convert cbr-to-cbz only. Since the logic is already in place it would be a pretty simple matter of adding that option to the GUI and doing a quick code piece for it.

toddos commented 8 years ago

I think the last option would be ideal. I'm not interested in tagging metadata, but if there's an already-existing and supported method of cbr-to-cbz then I'd prefer to use that rather than roll my own. I'm not anti-metadata, so I'll look at that as a proper solution for my needs.

evilhero commented 8 years ago

cbr-to-cbz conversion only will be available in the next development commit: metatagging Note that it's a toggle option, so if it's enabled - the remaining options disappear. If it's unchecked then the remaining options remain for selection.

Also, the pre-script option is working again where it will scan items after a pre-script run (like previously was the case) - so now you have the option to either use the built-in method for conversion or you can still use your pre-script to convert things.