search

clinton-hall / nzbToMedia

Provides NZB and Torrent postprocessing To CouchPotatoServer, SickBeard/SickRage, HeadPhones, Mylar and Gamez
GNU General Public License v3.0
673 stars 176 forks source link

TorrentToMedia | UserScript won't execute if download single file, not in directory #262

Closed ghost closed 10 years ago

ghost commented 10 years ago

Wondering if there is any way to fix this?

I'm using "other" and "hard".

TorrentToMedia works perfect with downloads sorted in their own directory by name. However, if the torrent just downloaded a single file (iso) as an example, and didn't download to a directory, nothing gets executed. Appears it expects a directory path to kick off the linking and script.

I can reproduce this single file behavior with this torrent: http://cdimage.debian.org/debian-cd/7.4.0/amd64/bt-cd/debian-7.4.0-amd64-netinst.iso.torrent

Everything works great with a directory torrent, such as: http://www.slackware.com/torrents/slackware-14.1-install-d1.torrent

Happens rarely that I come across an individual file.

clinton-hall commented 10 years ago

Are you able to open postprocess.log (same folder as TorrentToMedia.py) and post the full debug log of the download of http://cdimage.debian.org/debian-cd/7.4.0/amd64/bt-cd/debian-7.4.0-amd64-netinst.iso.torrent?

In theory this should work... so I just need to trace through what output directory is set to, where files are linked (if at all) and what is passed to user-script etc...

ghost commented 10 years ago

Running latest dev.

00:08:40|INFO TorrentToMedia V9.2 00:08:40|INFO MAIN: Loading config from /home/user/.scripts/nzbToMedia/autoP rocessMedia.cfg 00:08:40|DEBUG arg 0 is: /home/user/.scripts/nzbToMedia/TorrentToMedia.py 00:08:40|DEBUG arg 1 is: /torrent/download/manual/debian-7.4.0-amd64-net inst.iso 00:08:40|DEBUG arg 2 is: debian-7.4.0-amd64-netinst.iso 00:08:40|DEBUG arg 3 is: manual 00:08:40|DEBUG arg 4 is: 32FBA9C3C96BF060D195D4D454009D6BACAAF577 00:08:40|DEBUG MAIN: Received Directory: /torrent/download/manual/debian -7.4.0-amd64-netinst.iso | Name: | Category: 00:08:40|DEBUG SEARCH: Found Category: manual in directory structure 00:08:40|INFO SEARCH: Determined Category to be: manual 00:08:40|INFO SEARCH: Found torrent directory /torrent/download/manual/ debian-7.4.0-amd64-netinst.iso in category directory /torrent/download/man ual 00:08:40|INFO MAIN: Download is a directory 00:08:40|INFO MAIN: Output directory set to: /torrent/process/manual/de bian-7.4.0-amd64-netinst.iso 00:08:40|DEBUG MAIN: Scanning files in directory: /torrent/download/manu al/debian-7.4.0-amd64-netinst.iso 00:08:40|INFO FLATTEN: Flattening directory: /torrent/process/manual/de bian-7.4.0-amd64-netinst.iso 00:08:40|INFO REMOVER: Removing empty folders in: /torrent/process/manu al/debian-7.4.0-amd64-netinst.iso 00:08:40|INFO MAIN: Processing user script /home/user/.scripts/uncat.sh. 00:10:40|INFO All files have been processed. Cleaning outputDirectory / torrent/process/manual/debian-7.4.0-amd64-netinst.iso

Doesn't even attempt to run the script against the filename, as is seen in a successful run from this debug output:

00:58:49|INFO MAIN: Processing user script /home/user/.scripts/uncat.sh. 00:58:49|INFO Running script /home/user/.scripts/uncat.sh /torrent/pr ocess/manual/slackware-14.1-iso/slackware-14.1-install-d1.iso.md5 on file /torrent/process/manual/slackware-14.1-iso/slackware-14.1-install-d1.iso.md5. 00:58:49|INFO UserScript /home/user/.scripts/uncat.sh was successfull 00:58:49|INFO Running script /home/user/.scripts/uncat.sh /torrent/pr ocess/manual/slackware-14.1-iso/slackware-14.1-install-d1.iso.txt on file /torrent/process/manual/slackware-14.1-iso/slackware-14.1-install-d1.iso.txt. 00:58:49|INFO UserScript /home/user/.scripts/uncat.sh was successfull 00:58:49|INFO Running script /home/user/.scripts/uncat.sh /torrent/pr ocess/manual/slackware-14.1-iso/slackware-14.1-install-d1.iso on file /tor rent/process/manual/slackware-14.1-iso/slackware-14.1-install-d1.iso. 00:59:19|INFO UserScript /home/user/.scripts/uncat.sh was successfull 00:59:19|INFO Running script /home/user/.scripts/uncat.sh /torrent/pr ocess/manual/slackware-14.1-iso/slackware-14.1-install-d1.iso.asc on file /torrent/process/manual/slackware-14.1-iso/slackware-14.1-install-d1.iso.asc. 00:59:19|INFO UserScript /home/user/.scripts/uncat.sh was successfull 01:01:19|INFO All files have been processed. Cleaning outputDirectory / torrent/process/manual/slackware-14.1-iso 01:01:19|INFO MAIN: All done.

autoProcessMedia.cfg sections:

[Torrent] clientAgent = other useLink = hard outputDirectory = /torrent/process categories = couchpotato,sickbeard,manual,movies,music,tv uTorrentWEBui = uTorrentUSR = uTorrentPWD = TransmissionHost = TransmissionPort = TransmissionUSR = TransmissionPWD = DelugeHost = DelugePort = DelugeUSR = DelugePWD = deleteOriginal = 0

[UserScript] user_script_categories = manual,movies,music,tv user_script_mediaExtensions = ALL user_script_path = /home/user/.scripts/uncat.sh user_script_param = FP user_script_successCodes = 0 user_script_clean = 1 delay = 120

clinton-hall commented 10 years ago

Wow. I see what is happening, but I am surprised.

os.path.isdir(/torrent/download/manual/debian-7.4.0-amd64-netinst.iso)

Is returning true.... So python thinks this is a directory, but then can't find any files inside to be linked etc...

I'm not sure if this is actually python, or Linux, that treats .iso as a dir, but I'll need to add a specific check for .iso files.

I'll do some tests and try to get something in the next few days.

ghost commented 10 years ago

Good catch. It seems you are right on. Some other ISO's came in and hit the script and it ignored them.

ghost commented 10 years ago

Adding to this, here is another wrinkle...

Other and Hard.

Hard Linking doesn't seem to be creating the path structure on the destination. It's flattening the structure in the linked destination for processing. UserScript properly runs on those destination links, but the structure has already been flattened.

Example:

13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00017.mpls to /torrent/process/movies/Blah.0000.Blah.1-ABC/00017.mpls

Debug (shortened a bit for readability):

13:15:30|INFO TorrentToMedia V9.2 13:15:30|INFO MAIN: Loading config from /home/user/.scripts/nzbToMedia/autoProcessMedia.cfg 13:15:30|DEBUG arg 0 is: /home/user/.scripts/nzbToMedia/TorrentToMedia.py 13:15:30|DEBUG arg 1 is: /torrent/download/movies/Blah.0000.Blah.1-ABC 13:15:30|DEBUG arg 2 is: Blah.0000.Blah.1-ABC 13:15:30|DEBUG arg 3 is: movies 13:15:30|DEBUG arg 4 is: 1234567890 13:15:30|DEBUG MAIN: Received Directory: /torrent/download/movies/Blah.0000.Blah.1-ABC | Name: | Category: 13:15:30|DEBUG SEARCH: Found Category: movies in directory structure 13:15:30|INFO SEARCH: Determined Category to be: movies 13:15:30|INFO SEARCH: Found torrent directory /torrent/download/movies/Blah.0000.Blah.1-ABC in category directory /torrent/download 13:15:30|INFO MAIN: Download is a directory 13:15:30|INFO MAIN: Output directory set to: /torrent/process/movies/Blah.0000.Blah.1-ABC 13:15:30|DEBUG MAIN: Scanning files in directory: /torrent/download/movies/Blah.0000.Blah.1-ABC 13:15:30|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/disc.inf for category movies 13:15:30|INFO CREATE DESTINATION: Creating destination folder: /torrent/process/movies/Blah.0000.Blah.1-ABC 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/disc.inf to /torrent/process/movies/Blah.0000.Blah.1-ABC/disc.inf 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/MovieObject.bdmv for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/MovieObject.bdmv to /torrent/process/movies/Blah.0000.Blah.1-ABC/MovieObject.bdmv 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/index.bdmv for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/index.bdmv to /torrent/process/movies/Blah.0000.Blah.1-ABC/index.bdmv 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00055.mpls for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00055.mpls to /torrent/process/movies/Blah.0000.Blah.1-ABC/00055.mpls 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00204.mpls for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00204.mpls to /torrent/process/movies/Blah.0000.Blah.1-ABC/00204.mpls 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00209.mpls for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00209.mpls to /torrent/process/movies/Blah.0000.Blah.1-ABC/00209.mpls 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00215.mpls for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00215.mpls to /torrent/process/movies/Blah.0000.Blah.1-ABC/00215.mpls 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00014.mpls for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00014.mpls to /torrent/process/movies/Blah.0000.Blah.1-ABC/00014.mpls 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00062.mpls for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/PLAYLIST/00062.mpls to /torrent/process/movies/Blah.0000.Blah.1-ABC/00062.mpls 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/bdmt_por.xml for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/bdmt_por.xml to /torrent/process/movies/Blah.0000.Blah.1-ABC/bdmt_por.xml 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/bdmt_spa.xml for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/bdmt_spa.xml to /torrent/process/movies/Blah.0000.Blah.1-ABC/bdmt_spa.xml 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/bdmt_fra.xml for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/bdmt_fra.xml to /torrent/process/movies/Blah.0000.Blah.1-ABC/bdmt_fra.xml 13:15:31|INFO MAIN: Found metadata file .jpg for file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/movie_metadata416.jpg 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/movie_metadata416.jpg to /torrent/process/movies/Blah.0000.Blah.1-ABC/movie_metadata416.jpg 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/bdmt_eng.xml for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/bdmt_eng.xml to /torrent/process/movies/Blah.0000.Blah.1-ABC/bdmt_eng.xml 13:15:31|INFO MAIN: Found metadata file .jpg for file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/movie_metadata640.jpg 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/META/DL/movie_metadata640.jpg to /torrent/process/movies/Blah.0000.Blah.1-ABC/movie_metadata640.jpg 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/MovieObject.bdmv for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/index.bdmv for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00207.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00210.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00015.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00218.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00212.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00205.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00055.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00204.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00209.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00215.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00014.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/PLAYLIST/00062.mpls for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00111.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00111.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00111.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00093.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00093.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00093.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00013.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00013.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00013.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00125.clpi for category movies 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00098.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00098.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00098.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00116.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00116.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00116.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00130.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00130.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00130.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00128.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00128.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00128.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00002.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00002.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00002.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00015.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00015.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00015.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00008.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00008.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00008.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00127.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00127.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00127.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00115.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00115.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00115.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00118.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00118.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00118.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00112.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00112.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00112.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00006.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00006.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00006.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00018.clpi for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/BACKUP/CLIPINF/00018.clpi to /torrent/process/movies/Blah.0000.Blah.1-ABC/00018.clpi 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/CLIPINF/00111.clpi for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/CLIPINF/00093.clpi for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/CLIPINF/00013.clpi for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/CLIPINF/00125.clpi for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/CLIPINF/00131.clpi for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/CLIPINF/00004.clpi for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/CLIPINF/00113.clpi for category movies 13:15:31|INFO COPYLINK: target file already exists. Nothing to be done 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00014.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00014.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00014.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00012.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00012.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00012.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00114.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00114.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00114.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00093.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00093.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00093.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00005.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00005.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00005.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00121.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00121.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00121.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00017.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00017.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00017.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00112.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00112.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00112.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00004.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00004.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00004.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00009.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00009.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00009.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00003.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00003.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00003.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00117.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00117.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00117.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00015.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00015.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00015.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00131.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00131.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00131.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00118.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00118.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00118.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00115.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00115.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00115.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00092.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00092.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00092.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00122.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00122.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00122.m2ts 13:15:31|INFO MAIN: Found file /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00123.m2ts for category movies 13:15:31|INFO COPYLINK: Hard linking /torrent/download/movies/Blah.0000.Blah.1-ABC/BDMV/STREAM/00123.m2ts to /torrent/process/movies/Blah.0000.Blah.1-ABC/00123.m2ts 13:15:31|INFO FLATTEN: Flattening directory: /torrent/process/movies/Blah.0000.Blah.1-ABC 13:15:31|INFO REMOVER: Removing empty folders in: /torrent/process/movies/Blah.0000.Blah.1-ABC 13:15:31|INFO MAIN: Processing user script /home/user/.scripts/uncat.sh. 13:15:31|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00111.clpi on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00111.clpi. 13:15:31|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:15:31|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00093.clpi on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00093.clpi. 13:15:31|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:15:31|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00207.mpls on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00207.mpls. 13:15:31|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:15:31|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00210.mpls on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00210.mpls. 13:15:31|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:15:31|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00008.m2ts on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00008.m2ts. 13:15:32|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:15:32|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00015.mpls on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00015.mpls. 13:15:32|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:15:32|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00007.m2ts on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00007.m2ts. 13:15:32|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:15:32|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00013.clpi on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00013.clpi. 13:15:32|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:15:32|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00217.mpls on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00217.mpls. 13:15:32|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:32:14|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00013.mpls on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00013.mpls. 13:32:14|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:32:14|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00092.clpi on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00092.clpi. 13:32:15|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:32:15|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00116.m2ts on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00116.m2ts. 13:33:07|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:33:07|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00111.m2ts on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00111.m2ts. 13:33:11|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:33:11|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/MKB_RO.inf on file /torrent/process/movies/Blah.0000.Blah.1-ABC/MKB_RO.inf. 13:33:11|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:33:11|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/ContentHash000.tbl on file /torrent/process/movies/Blah.0000.Blah.1-ABC/ContentHash000.tbl. 13:33:12|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:42:05|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00204.mpls on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00204.mpls. 13:42:05|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:42:05|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00006.clpi on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00006.clpi. 13:42:05|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:42:05|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00209.mpls on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00209.mpls. 13:42:05|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:42:05|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00215.mpls on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00215.mpls. 13:42:05|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:42:05|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00014.mpls on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00014.mpls. 13:42:05|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:42:05|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/movie_metadata640.jpg on file /torrent/process/movies/Blah.0000.Blah.1-ABC/movie_metadata640.jpg. 13:42:05|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:42:05|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00122.m2ts on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00122.m2ts. 13:43:21|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:43:21|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00062.mpls on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00062.mpls. 13:43:21|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:43:21|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00018.clpi on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00018.clpi. 13:43:21|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:43:21|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/00123.m2ts on file /torrent/process/movies/Blah.0000.Blah.1-ABC/00123.m2ts. 13:44:25|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:44:25|INFO Running script /home/user/.scripts/uncat.sh /torrent/process/movies/Blah.0000.Blah.1-ABC/mcmf.xml on file /torrent/process/movies/Blah.0000.Blah.1-ABC/mcmf.xml. 13:44:25|INFO UserScript /home/user/.scripts/uncat.sh was successfull 13:46:25|INFO All files have been processed. Cleaning outputDirectory /torrent/process/movies/Blah.0000.Blah.1-ABC 13:46:25|INFO MAIN: All done.

clinton-hall commented 10 years ago

Very interesting. So the full structure of the download is sometimes important?

(keep in mind that originally these scripts started for processing a single episode of TV or a single movie at a time)...

By default I was only moving the wanted files into the destination, and yes, flattening was intended for this... so just how does this cause problems with the user-script?

I can't replicate the .iso issue.

Can you please verify that /torrent/download/manual/debian-7.4.0-amd64-netinst.iso is in fact an iso file on your system (and that there is no directory /torrent/download/manual/debian-7.4.0-amd64-netinst.iso)

Then can you run python and do the following?

import os
os.path.isdir(/torrent/download/manual/debian-7.4.0-amd64-netinst.iso) 
os.path.isfile(/torrent/download/manual/debian-7.4.0-amd64-netinst.iso)

Post the output here.

Also, what version of Python and what OS are you using?

ghost commented 10 years ago

Ubuntu 13.10 Python 2.7.5+ (default, Sep 19 2013, 13:48:49) [GCC 4.8.1] on linux2

user@ubuntu:/torrent/download/manual$ file debian-7.4.0-amd64-netinst.iso 
debian-7.4.0-amd64-netinst.iso: # ISO 9660 CD-ROM filesystem data 'Debian 7.4.0 amd64 1            ' (bootable)
Python 2.7.5+ (default, Sep 19 2013, 13:48:49) 
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import os

>>> os.path.isdir(/torrent/download/manual/debian-7.4.0-amd64-netinst.iso)
  File "<stdin>", line 1
    os.path.isdir(/torrent/download/manual/debian-7.4.0-amd64-netinst.iso)
                        ^
SyntaxError: invalid syntax

>>> os.path.isfile(/torrent/download/manual/debian-7.4.0-amd64-netinst.iso)
  File "<stdin>", line 1
    os.path.isfile(/torrent/download/manual/debian-7.4.0-amd64-netinst.iso)
                   ^
SyntaxError: invalid syntax

Needs quotes it seems:

>>> os.path.isfile("/usr/bin/git")
True

>>> os.path.isfile("/torrent/download/manual/debian-7.4.0-amd64-netinst.iso")
True

>>> os.path.isdir("/torrent/download/manual/debian-7.4.0-amd64-netinst.iso")
False
clinton-hall commented 10 years ago

ah... quotes... sorry.

So it isn't seeing .iso as directory... when the directory given from the downloader was /torrent/download/manual/debian-7.4.0-amd64-netinst.iso and we then work down to find /torrent/download/manual as the category directory, it just ASSUMED debian-7.4.0-amd64-netinst.iso was a directory.

I now test for this (see the last commit) and actually verify this to be a file, before then continuing.

This should fix the single file download issue. But not sure what to do with the keeping directory structure. Perhaps a switch for certain categories to replicate structure?

This will be a bit more work.

ghost commented 10 years ago

Testing latest commit.

Looks like it's properly detecting the single file.. but syntax error:

user@ubuntu:~/.scripts/nzbToMedia$ python TorrentToMedia.py /torrent/download/manual/debian-7.4.0-amd64-netinst.iso debian-7.4.0-amd64-netinst.iso manual 32FBA9C3C96BF060D195D4D454009D6BACAAF577
Traceback (most recent call last):
  File "TorrentToMedia.py", line 16, in <module>
    import extractor.extractor as extractor
  File "/home/user/.scripts/nzbToMedia/extractor/extractor.py", line 7, in <module>
    from autoProcess.nzbToMediaUtil import create_destination
  File "/home/user/.scripts/nzbToMedia/autoProcess/nzbToMediaUtil.py", line 104
    inputDirectory = os.path.normpath(categorySearch[0]):
                                                        ^
SyntaxError: invalid syntax
ghost commented 10 years ago

thinking out loud...

your comments about original intent of the userscript file process is understood. ideally i'm trying to use TorrentToMedia to process everything, as it's very efficient and smart about post-processing. because i can define directories that i want to be hard link, and others to exclude (torrent client not so smart), i can be particular about what goes where. CP and SB get handled properly, and then only certain folders get processed by a script. no need to hard link or deal with the other directories i don't define and care about.

Keeping directory structure is useful.. ideally with keeping the hard linking support in place. i think that means having to pre-create the directory structure first?

the primary issue is with blu-ray rips (non-ISO) from a torrent. the structure can be important and ideally the entire directory gets moved to a new location by a script. or music with cd1/cd2, etc..

a switch to enable or disable would be ideal for everyone. they can choose.

or...

instead of running userscript for every file; have an option to run it once and pass the full directory? you wouldn't have to hard link, potentially, with this option, we could handle it in our own script.

or, if variable = DN, if you could just pass DN once, that would work. a script can take it from there, rsync or cp -R the entire DN.

clinton-hall commented 10 years ago

Ok... fixed the syntaxt error.

Also added 2 new options

[Torrent] noFlatten = manual,videos ... etc

This is a list of categories you don't want flattened. You can use "UNCAT" here, but not ALL

[UserScript] user_script_runOnce = 1

This will only let the userscript run once (presumably you will only be passing DN and not FN etc).

I have not tested these yet (just ran out of time after scripting this in), so there will likely be more syntaxt errors ;)

ghost commented 10 years ago

with latest commit won't run at all.. code not working.

File "./TorrentToMedia.py", line 343 if user_script_runOnce == 1 and num_files > 1: we have already run once, so just continue to get number of files.

going to revert one commit for now.

ghost commented 10 years ago

with syntax correction, still issues.:

Traceback (most recent call last):
  File "./TorrentToMedia.py", line 16, in <module>
    import extractor.extractor as extractor
  File "/home/user/.scripts/nzbToMedia/extractor/extractor.py", line 7, in <module>
    from autoProcess.nzbToMediaUtil import create_destination
  File "/home/user/.scripts/nzbToMedia/autoProcess/nzbToMediaUtil.py", line 107
    Logger.info("SEARCH: Setting input directory to %s" and checking for files by date modified., categorySearch[0])
                                                                                ^
SyntaxError: invalid syntax
clinton-hall commented 10 years ago

Sorry.... that is why I shouldn't try and add new features in my lunch break.

Try updating to latest and try again.

ghost commented 10 years ago

that was it. thank you for getting this together so quick. you need a donation page so we can help you out for all this work. (edit: found your page)

so, your implementation of single file is better than expected. creating a destination directory is ideal. it works perfect.

we can close the original problem, handling of single file downloads is working perfectly.

ghost commented 10 years ago

following on to this.. for noFlatten.. will it create the directory structure on the destination if i leave user_script_runOnce = 0? will it build the structure on the destination and still send every file to userscript individually?

ghost commented 10 years ago

one of these downloads with directories happen to come in...

[Torrent] clientAgent = other useLink = hard outputDirectory = /cache/torrent/process categories = couchpotato,sickbeard,manual,movies,music,tv noFlatten = manual,movies,music,tv

looks like it attempted to create the directory structure, but the path variable is not correct. it is not appending the outputdirectory to outputdestination:

00:36:56|INFO    ====================
00:36:56|INFO    TorrentToMedia V9.2
00:36:56|INFO    MAIN: Loading config from /home/user/.scripts/nzbToMedia/autoProcessMedia.cfg
00:36:56|DEBUG   arg 0 is: /home/user/.scripts/nzbToMedia/TorrentToMedia.py
00:36:56|DEBUG   arg 1 is: /cache/torrent/download/movies/BLAH
00:36:56|DEBUG   arg 2 is: BLAH
00:36:56|DEBUG   arg 3 is: movies
00:36:56|DEBUG   arg 4 is: 01234567890
00:36:56|DEBUG   MAIN: Received Directory: /cache/torrent/download/movies/BLAH | Name:  | Category: 
00:36:56|DEBUG   SEARCH: Found Category: movies in directory structure
00:36:56|INFO    SEARCH: Determined Category to be: movies
00:36:56|INFO    SEARCH: Found torrent name: BLAH
00:36:56|INFO    SEARCH: Found torrent directory /cache/torrent/download/movies/BLAH in category directory /cache/torrent/download/movies
00:36:56|INFO    MAIN: Download is a directory
00:36:56|INFO    MAIN: Output directory set to: /cache/torrent/process/movies/BLAH
00:36:56|DEBUG   MAIN: Scanning files in directory: /cache/torrent/download/movies/BLAH
00:36:56|DEBUG   MAIN: Setting outputDestination to /SUBDIR to preserve folder structure
00:36:56|INFO    MAIN: Found file /cache/torrent/download/movies/BLAH/SUBDIR/some.file for category movies
00:36:56|INFO    CREATE DESTINATION: Creating destination folder: /SUBDIR
00:36:56|ERROR   CREATE DESTINATION: Not possible to create destination folder. Exiting
Traceback (most recent call last):
  File "/home/user/.scripts/nzbToMedia/autoProcess/nzbToMediaUtil.py", line 36, in create_destination
    os.makedirs(outputDestination)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/SUBDIR'
clinton-hall commented 10 years ago

following on to this.. for noFlatten.. will it create the directory structure on the destination if i leave user_script_runOnce = 0? will it build the structure on the destination and still send every file to userscript individually?

Yes, it will build the structure on the destination and then send every file to userscript individually. So these two parameters (noFlatten, user _script_runOnce) work individually to achieve different things.

I'm looking into the failure in creating the directory structure at destination... Sorry about this.

clinton-hall commented 10 years ago

Ah..... My method was right, but since the extra path we are adding is. /SUBDIR when we join Output and /SUBDIR the leading "/" sets this back to root...

In other words. I need to strip the leading "/" from /SUBDIR before trying to join this onto outputDirectory....

More to come.

clinton-hall commented 10 years ago

Hopefully this does it... let me know how it goes.

This last change was done with a beer in hand ;)

ghost commented 10 years ago

Seems we are back to where we started with individual file. though, rtorrent properly detecting additional information now.

update 1: not any of the UserScript settings, DN or FP or runOnce. tested every way, no difference.

update 2: if i set clientAgent=other, it seems to begin to set the link into a directory part, then fails.

./TorrentToMedia.py /cache/torrent/download/manual/debian-update-7.4.0-amd64-CD-9.iso debian-update-7.4.0-amd64-CD-9.iso manual BDF9E9EF931CD1A475FBAF3E6FA54B6E61963AA6

19:27:31|INFO    ====================
19:27:31|INFO    TorrentToMedia V9.2
19:27:31|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
19:27:31|INFO    SEARCH: Determined Category to be: manual
19:27:31|INFO    SEARCH: Found torrent name: debian-update-7.4.0-amd64-CD-9.iso
19:27:31|INFO    SEARCH: /cache/torrent/download/manual/debian-update-7.4.0-amd64-CD-9.iso is a file, not a directory.
19:27:31|INFO    SEARCH: Setting input directory to /cache/torrent/download/manual
19:27:31|INFO    MAIN: Download is not a directory
19:27:31|INFO    MAIN: Output directory set to: /cache/torrent/process/manual/debian-update-7.4.0-amd64-CD-9
Traceback (most recent call last):
  File "./TorrentToMedia.py", line 496, in <module>
    main(inputDirectory, inputName, inputCategory, inputHash, inputID)
  File "./TorrentToMedia.py", line 137, in main
    if newDir[0] == "/":
IndexError: string index out of range

[Torrent] clientAgent = rtorrent useLink = hard outputDirectory = /cache/torrent/process categories = couchpotato,sickbeard,manual,movies,music,tv noFlatten = manual,movies,music,tv uTorrentWEBui = http://localhost:8090/gui/ uTorrentUSR = your username uTorrentPWD = your password TransmissionHost = localhost TransmissionPort = 8084 TransmissionUSR = your username TransmissionPWD = your password DelugeHost = DelugePort = DelugeUSR = DelugePWD = deleteOriginal = 0

[UserScript] user_script_categories = manual,movies,music,tv user_script_mediaExtensions = ALL user_script_path = /home/user/.scripts/uncat.sh user_script_param = DN user_script_runOnce = 1 user_script_successCodes = 0 user_script_clean = 1 delay = 120

19:18:26|INFO    ====================
19:18:26|INFO    TorrentToMedia V9.2
19:18:26|INFO    MAIN: Loading config from /home/user/.scripts/nzbToMedia/autoProcessMedia.cfg
19:18:26|DEBUG   arg 0 is: /home/user/.scripts/nzbToMedia/TorrentToMedia.py
19:18:26|DEBUG   arg 1 is: /cache/torrent/download/manual/debian-update-7.4.0-amd64-CD-9.iso
19:18:26|DEBUG   arg 2 is: debian-update-7.4.0-amd64-CD-9.iso
19:18:26|DEBUG   arg 3 is: manual
19:18:26|DEBUG   arg 4 is: BDF9E9EF931CD1A475FBAF3E6FA54B6E61963AA6
19:18:26|DEBUG   MAIN: Received Directory: /cache/torrent/download/manual/debian-update-7.4.0-amd64-CD-9.iso | Name: debian-update-7.4.0-amd64-CD-9.iso | Category: manual
19:18:26|INFO    SEARCH: Files appear to be in their own directory
19:18:26|INFO    MAIN: Download is a directory
19:18:26|INFO    MAIN: Output directory set to: /cache/torrent/process/manual/debian-update-7.4.0-amd64-CD-9.iso
19:18:26|DEBUG   MAIN: Scanning files in directory: /cache/torrent/download/manual/debian-update-7.4.0-amd64-CD-9.iso
19:18:26|INFO    MAIN: Processing user script /home/user/.scripts/uncat.sh.
ghost commented 10 years ago

after that commit, with "other":

[Torrent] clientAgent = other useLink = hard

19:48:34|INFO    ====================
19:48:34|INFO    TorrentToMedia V9.2
19:48:34|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
19:48:34|INFO    SEARCH: Determined Category to be: manual
19:48:34|INFO    SEARCH: Found torrent name: debian-update-7.4.0-amd64-CD-9.iso
19:48:34|INFO    SEARCH: /cache/torrent/download/manual/debian-update-7.4.0-amd64-CD-9.iso is a file, not a directory.
19:48:34|INFO    SEARCH: Setting input directory to /cache/torrent/download/manual
19:48:34|INFO    MAIN: Download is not a directory
19:48:34|INFO    MAIN: Output directory set to: /cache/torrent/process/manual/debian-update-7.4.0-amd64-CD-9
19:48:34|INFO    MAIN: Found video file .iso in /cache/torrent/download/manual/debian-update-7.4.0-amd64-CD-9.iso
19:48:34|ERROR   MAIN: Failed to link file: debian-update-7.4.0-amd64-CD-9.iso
Traceback (most recent call last):
  File "./TorrentToMedia.py", line 173, in main
    copy_link(filePath, targetDirectory, useLink, outputDestination)
UnboundLocalError: local variable 'useLink' referenced before assignment
19:48:34|INFO    MAIN: Processing user script /home/media/.scripts/uncat.sh.
^CTraceback (most recent call last):
  File "./TorrentToMedia.py", line 496, in <module>
    main(inputDirectory, inputName, inputCategory, inputHash, inputID)
  File "./TorrentToMedia.py", line 250, in main
    result = external_script(outputDestination)
  File "./TorrentToMedia.py", line 389, in external_script
    time.sleep(user_delay)
clinton-hall commented 10 years ago

useLink was defined, but maybe some issue with referencing this inside different work space?... really not sure about this

clinton-hall commented 10 years ago

this last fix here should allow you to set clientAgent = rtorrent again... essentially when the torrent name was passed (which didn't happen for other) there was a match to the directory name that said this was a unique directory... I (hopefully) applied the same fix as applied elsewhere, where this path is tested to ensure it is a directory...

You sure have found a lot of neat new combinations and done some great testing... sorry this has taken so long to iron it all out.

That "local variable 'useLink' referenced before assignment" still puzzles me, but hopefully the previous fix above fixes that.

ghost commented 10 years ago

rtorrent still not working.

20:17:36|INFO    ====================
20:17:36|INFO    TorrentToMedia V9.2
20:17:36|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
20:17:36|INFO    MAIN: Download is a directory
20:17:36|INFO    MAIN: Output directory set to: /cache/torrent/process/manual/debian-update-7.4.0-amd64-CD-9.iso
20:17:36|INFO    MAIN: Processing user script /home/media/.scripts/uncat.sh.

however, other is working properly. for single file and for directory.

thanks for working this out with me. i think this is some pretty strong functionality for everyone!

clinton-hall commented 10 years ago

agreed... some good functionality here... thanks for helping resolve these issues... so hard to code and test situations when i don't have the same config to test on.

This last fix "should" resolve the issues with rtorrent downloader...

ghost commented 10 years ago

just a note:

when you set to DN and runOnce it passes the DN and then exits with success; doesn't wait for completion. it does keep processing script, however...

clinton-hall commented 10 years ago

odd... it should handle this the same.

Depending on what the script does, it might not wait for that script to finish, but it then should wait for user_delay seconds, and then check to see if any of the files have moved etc before exiting...

ghost commented 10 years ago

ahh.. DN, runOnce.. if not flattened, just sends first file's dirname, which won't cover everything. i'm going back to FP in the meantime to see if I can get this sorted out.

you are correct about delay, it just doesn't wait for the exit 0 before saying success.

clinton-hall commented 10 years ago

I had expected that the first file's dirname would be the root dirname, but this isn't true if there are no files in the base directory you want processed.

I can change this so that if runOnce and DN, DN is replaces with outptDirectory?

ghost commented 10 years ago

i think that is ideal.. ultimately you'll just be using rsync in a situation like this. or some recursive cp or mv.

clinton-hall commented 10 years ago

one of these days i might do some work... but until then, let me know what else to add ;)

ghost commented 10 years ago

so, rtorrent still not properly grabbing single file; but directory works perfect.

as for DN and runOnce.. that is working pretty nice. with single file or directory. really nice to puts single file into a directory for processing; i think that is ideal.

this is the command I get the script to run:

/usr/bin/rsync -acq /cache/torrent/process/movies/BLAH /storage/Working/Download/movies

my simple bash script:

#!/bin/bash

SOURCE="/cache/torrent/process"
DEST="/storage/Working/Download"
CPCMD="/usr/bin/rsync -acq"

# Argument Check
if [ -z "$1" ]; then
    exit 1
fi

# DDN = Destination Dir Path (assumes like dir structure)
DEST_DIR=`dirname $(echo "${1}" | sed "s|${SOURCE}|${DEST}|")`
BASENAME=`basename "${1}"`
FINAL_DEST="${DEST_DIR}/${BASENAME}"

# Perform Transfer
$CPCMD "${1}" "${DEST_DIR}"

# Confirm destination exists, then exit 0
if [ -e "${FINAL_DEST}" ]; then
    exit 0
  else
    exit 1
fi
ghost commented 10 years ago

DN waiting for script to exit.. i think that was my issue. all working perfectly right now with other. rtorrent works great for dir, no so well (original issues) with single file. progress!

clinton-hall commented 10 years ago

have you got a log output for rtorrent with single directory? I'm sure there is probably just one more step where it thinks it is a directory...

ghost commented 10 years ago 
21:18:55|INFO    ====================
21:18:55|INFO    TorrentToMedia V9.2
21:18:55|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
21:18:55|DEBUG   arg 0 is: ./TorrentToMedia.py
21:18:55|DEBUG   arg 1 is: /cache/torrent/download/manual/test.out
21:18:55|DEBUG   arg 2 is: test
21:18:55|DEBUG   arg 3 is: manual
21:18:55|DEBUG   arg 4 is: 
21:18:55|DEBUG   MAIN: Received Directory: /cache/torrent/download/manual/test.out | Name: test | Category: manual
21:18:55|DEBUG   SEARCH: Found Category: manual in directory structure
21:18:55|INFO    MAIN: Download is not a directory
21:18:55|INFO    MAIN: Output directory set to: /cache/torrent/process/manual/test
21:18:55|DEBUG   MAIN: Scanning files in directory: /cache/torrent/download/manual/test.out
21:18:55|INFO    MAIN: Processing user script /home/media/.scripts/uncat.sh.
21:20:33|INFO    ====================
21:20:33|INFO    TorrentToMedia V9.2
21:20:33|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
21:20:33|DEBUG   arg 0 is: ./TorrentToMedia.py
21:20:33|DEBUG   arg 1 is: /cache/torrent/download/manual/test.out
21:20:33|DEBUG   arg 2 is: test
21:20:33|DEBUG   arg 3 is: manual
21:20:33|DEBUG   arg 4 is: 
21:20:33|DEBUG   MAIN: Received Directory: /cache/torrent/download/manual/test.out | Name: test | Category: manual
21:20:33|DEBUG   SEARCH: Found Category: manual in directory structure
21:20:33|INFO    MAIN: Download is not a directory
21:20:33|INFO    MAIN: Output directory set to: /cache/torrent/process/manual/test
21:20:33|DEBUG   MAIN: Scanning files in directory: /cache/torrent/download/manual/test.out
21:20:33|INFO    MAIN: Processing user script /home/media/.scripts/uncat.sh.
media@download:~/.scripts/nzbToMedia$ ./TorrentToMedia.py /cache/torrent/download/manual/test.out test.out manual ""
22:27:54|INFO    ====================
22:27:54|INFO    TorrentToMedia V9.2
22:27:54|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
22:27:54|INFO    MAIN: Download is not a directory
22:27:54|INFO    MAIN: Output directory set to: /cache/torrent/process/manual/test
22:27:54|INFO    MAIN: Processing user script /home/media/.scripts/uncat.sh.
media@download:~/.scripts/nzbToMedia$ ./TorrentToMedia.py /cache/torrent/download/manual/debian-update-7.4.0-amd64-CD-9.iso debian-update-7.4.0-amd64-CD-9 manual ""
22:29:19|INFO    ====================
22:29:19|INFO    TorrentToMedia V9.2
22:29:19|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
22:29:19|INFO    MAIN: Download is not a directory
22:29:19|INFO    MAIN: Output directory set to: /cache/torrent/process/manual/debian-update-7.4
22:29:19|INFO    MAIN: Processing user script /home/media/.scripts/uncat.sh.
clinton-hall commented 10 years ago

In theory this should catch any eventuality where the input directory is not a directory.... So should work for rtorrent.

ghost commented 10 years ago

interesting one now that I'm testing others:

23:29:02|INFO    ====================
23:29:02|INFO    TorrentToMedia V9.2
23:29:02|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
23:29:02|INFO    MAIN: Calling SickBeard's TPB branch to post-process: Blah.Blah.mkv
23:29:02|INFO    Loading config from ./autoProcessMedia.cfg
Traceback (most recent call last):
  File "./TorrentToMedia.py", line 500, in <module>
    main(inputDirectory, inputName, inputCategory, inputHash, inputID)
  File "./TorrentToMedia.py", line 43, in main
    result = autoProcessTV.processEpisode(inputDirectory, inputName, int(0))
  File "/home/media/.scripts/nzbToMedia/autoProcess/autoProcessTV.py", line 124, in processEpisode
    if not fork in SICKBEARD_TORRENT:
UnboundLocalError: local variable 'SICKBEARD_TORRENT' referenced before assignment

[SickBeard] sbCategory = sickbeard host = 127.0.0.1 port = 8081 username = password = web_root = /sickbeard ssl = 1 delay = 0 wait_for = 10 watch_dir = fork = TPB delete_failed = 1

ghost commented 10 years ago

rtorrent success! both single file and directory! :-)

well, almost.. this is a minor one. doesn't create full outputDirectory name, truncates it.

23:35:44|INFO    ====================
23:35:44|INFO    TorrentToMedia V9.2
23:35:44|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
23:35:44|INFO    MAIN: Download is not a directory
23:35:44|INFO    MAIN: Output directory set to: /cache/torrent/process/manual/debian-update-7.4
23:35:44|INFO    MAIN: Found video file .iso in /cache/torrent/download/manual/debian-update-7.4.0-amd64-CD-9.iso
23:35:44|INFO    COPYLINK: target file already exists. Nothing to be done
23:35:44|INFO    MAIN: Processing user script /home/media/.scripts/uncat.sh.
23:35:44|INFO    Running script  /home/media/.scripts/uncat.sh /cache/torrent/process/manual/debian-update-7.4 on file /cache/torrent/process/manual/debian-update-7.4/debian-update-7.4.0-amd64-CD-9.iso.
23:35:44|INFO    UserScript /home/media/.scripts/uncat.sh was successfull

should be: /cache/torrent/process/manual/debian-update-7.4.0-amd64-CD-9

but instead truncates to: /cache/torrent/process/manual/debian-update-7.4

clinton-hall commented 10 years ago

ok... SickBeard issue should be fixed, and I have tidied up that whole test for single file (do it once instead of putting it inside every if statement ;) )

the truncated outputdirectory is due to my use of SafeName... this removes character that may cause issues in some operating systems... this might be a bit over the top, but since it is only in the linked/copied folder, shouldn't be an issue... we can still pass the download name to userscript if needed?

thanks for all the testing you have done here... made it easy to make a lot of improvements :)

ghost commented 10 years ago

i think passing download name is cool. i can then do a mkdir first, easy enough. makes sense to keep SafeName in place. is there currently a variable for that; i didn't think there was? in the case of using runOnce and DN, the others like FN or FP wouldn't necessarily accomplish the end goal.

ghost commented 10 years ago

after latest commits, single file handling not working again?

23:16:11|INFO    ====================
23:16:11|INFO    TorrentToMedia V9.2
23:16:11|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
23:16:11|INFO    SEARCH: Could not find a unique torrent folder in the directory structure
23:16:11|INFO    SEARCH: The directory passed is the root directory for category movies
23:16:11|WARNING SEARCH: You should change settings to download torrents to their own directory if possible
23:16:11|INFO    SEARCH: We will try and determine which files to process, individually
23:16:11|INFO    MAIN: Download is a directory
23:16:11|INFO    MAIN: Output directory set to: /cache/torrent/process/movies/movies
23:16:11|INFO    MAIN: Processing user script /home/media/.scripts/uncat.sh.
clinton-hall commented 10 years ago

can you post the debug log for this last one? I can see what's not happening, but not why.

ghost commented 10 years ago

sorry about that.. this is actually very, very interesting... it literally went through every file / folder in the root of the source destination and scanned it, creating an outputDestination for each... it's 2200 lines long.. then script hangs...

here is a sample:

23:16:11|INFO    ====================
23:16:11|INFO    TorrentToMedia V9.2
23:16:11|INFO    MAIN: Loading config from ./autoProcessMedia.cfg
23:16:11|DEBUG   arg 0 is: ./TorrentToMedia.py
23:16:11|DEBUG   arg 1 is: /cache/torrent/download/movies/BLAH.iso
23:16:11|DEBUG   arg 2 is: BLAH.iso
23:16:11|DEBUG   arg 3 is: movies
23:16:11|DEBUG   arg 4 is: 123
23:16:11|DEBUG   MAIN: Received Directory: /cache/torrent/download/movies/Blah.iso | Name: Blah.iso | Category: movies
23:16:11|DEBUG   SEARCH: Found Category: movies in directory structure
23:16:11|INFO    SEARCH: Could not find a unique torrent folder in the directory structure
23:16:11|INFO    SEARCH: The directory passed is the root directory for category movies
23:16:11|WARNING SEARCH: You should change settings to download torrents to their own directory if possible
23:16:11|INFO    SEARCH: We will try and determine which files to process, individually
23:16:11|INFO    MAIN: Download is a directory
23:16:11|INFO    MAIN: Output directory set to: /cache/torrent/process/movies/movies
23:16:11|DEBUG   MAIN: Scanning files in directory: /cache/torrent/download/movies
23:16:11|DEBUG   MAIN: Setting outputDestination to /cache/torrent/process/movies/movies/ to preserve folder structure
23:16:11|DEBUG   MAIN: Looking for movies in: Blah1.iso
23:16:11|DEBUG   MAIN: Setting outputDestination to /cache/torrent/process/movies/movies/ to preserve folder structure
23:16:11|DEBUG   MAIN: Looking for movies in: Blah2.iso
23:16:11|DEBUG   MAIN: Setting outputDestination to /cache/torrent/process/movies/movies/ to preserve folder structure
23:16:11|DEBUG   MAIN: Looking for movies in: Blah3.iso
23:16:11|DEBUG   MAIN: Setting outputDestination to /cache/torrent/process/movies/movies/ to preserve folder structure
23:16:11|DEBUG   MAIN: Looking for movies in: Blah4.iso
23:16:11|DEBUG   MAIN: Setting outputDestination to /cache/torrent/process/movies/movies/ to preserve folder structure
23:16:11|DEBUG   MAIN: Looking for movies in: Blah.iso
23:16:11|DEBUG   MAIN: Setting outputDestination to /cache/torrent/process/movies/movies/ to preserve folder structure
23:16:11|DEBUG   MAIN: Looking for movies in: Blah5.iso
23:16:11|DEBUG   MAIN: Setting outputDestination to /cache/torrent/process/movies/movies/blah10 to preserve folder structure
23:16:11|DEBUG   MAIN: Looking for movies in: blah10.r39
[...]
23:16:11|DEBUG   MAIN: Setting outputDestination to /cache/torrent/process/movies/movies/Blah11/Sample to preserve folder structure
23:16:11|DEBUG   MAIN: Looking for movies in: blah11.sample.m2ts
23:16:11|INFO    MAIN: Processing user script /home/media/.scripts/uncat.sh.
clinton-hall commented 10 years ago

So stupid of me... I put an earlier test that if the inputDirectory is a file (not directory) we split into inputdirectory and inputname...

great in theory, but first I split input directory (ofource, the file name is now stripped off) and then I split to get file name, and the filename became "movies" which was the category!

I now just split this in one line... neater, and should give the right answer.

Sometimes a stupid little ordering issue can make life difficult!

clinton-hall commented 10 years ago

I also added more debug logging to tell us what inputs are passed AND what inputs are determined from parsing the directory structure.

clinton-hall commented 10 years ago

You can now add "TN" to userscript to pass through the TorrentName/DownloadName.

ghost commented 10 years ago

thank you. one more ask, just to clean things up even better. any chance to pass label/category?

clinton-hall commented 10 years ago

You can now add "TL" to userscript to pass through the Torrent Label/Category.

ghost commented 10 years ago

noFlatten, directory structure = working! single file = working! (creates safename directory) TN and TL = working!

thank you for extra logging. always helpful.