Dev branch

[nctiggy]

So, this is without doing much research into the code changes but it appears that the conversion is shooting the output directly into the destination directory (presumably to avoid a move action). This creates a few issues for me, but may not be big enough deals for the majority of script users. My destination is the "Automatically move to iTunes" folder. The concern is two fold, There have been conversion failures that have left empty mp4's in the folder and imported as movies into iTunes and secondly there is a risk that the file will be imported into iTunes between conversion and tagging. This has been evidenced by a couple films that have been imported with no cover or tagging (EDIT: Further validated with running dev branch with sabnzbd, TV shows have imported without being tagged). Firstly it would be nice to convert in-place to the same location as the source and then move afterwards. Second perhaps it would be valuable to offer a -log flag to direct the output to a log file, this would help with large batch jobs so we can see where and if there were issues along the way.

Short of all that longwindedness the manual script is running very well. Thoughts?

[nctiggy]

Trying to run the script on one of the already converted files (that didn't tag) spits this out:

craig@nzb:~/sickbeard_mp4_automator$ ./manual.py /home/craig/Movies/iTunes/Home\ Videos/New.Girl.S03E03.720p.HDTV.X264-DIMENSION.mp4 -silent /home/craig/Movies/iTunes/Home Videos/New.Girl.S03E03.720p.HDTV.X264-DIMENSION.mp4 detected for potential conversion - already correct format, skipping reprocessing Got here 2 Got here Error moving file to output directory /home/craig/Movies/iTunes/Home Videos/New.Girl.S03E03.720p.HDTV.X264-DIMENSION.mp4 and /home/craig/Movies/iTunes/Automatically Add to iTunes.localized/New.Girl.S03E03.720p.HDTV.X264-DIMENSION.mp4 are the same file Relocating MOOV atom to start of file Traceback (most recent call last): File "./manual.py", line 217, in main() File "./manual.py", line 195, in main processFile(path, tagdata) File "./manual.py", line 156, in processFile source.QTFS() File "/home/craig/sickbeard_mp4_automator/mkvtomp4.py", line 287, in QTFS processor.process(outputfile, tmp) File "/home/craig/sickbeard_mp4_automator/qtfaststart/processor.py", line 188, in process outfile = open(outfilename, "wb") IOError: [Errno 13] Permission denied: '/home/craig/Movies/iTunes/Home Videos/New.Girl.S03E03.720p.HDTV.X264-DIMENSION.mp4.tmp'

[mdhiggins]

Not sure why shutil.copy is giving you the following error: /home/craig/Movies/iTunes/Home Videos/New.Girl.S03E03.720p.HDTV.X264-DIMENSION.mp4 and /home/craig/Movies/iTunes/Automatically Add to iTunes.localized/New.Girl.S03E03.720p.HDTV.X264-DIMENSION.mp4 are the same file

Are you using virtualized directories or anything strange like that? There also seems to be a permissions issue.

EDIT: I'm unable to recreate your issue. I took an untagged mp4 and processed it, and the resulting file ended up in my output_dir that was set with the file properly tagged.

[mdhiggins]

As far as the output_dir issue you expressed above, I guess the only way to satisfy both situations is to add an option to process the new file in the same directory as the original. I'll add that

[nctiggy]

ok. I am just not sure, but in the dev branch when running sab under normal conditions TV is just not tagging at all.

       /home/craig/storage/complete/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION/the.amazing.race.2301.720p-dimension.sfv detected for potential conversion - file not in the correct format, ignoring

/home/craig/storage/complete/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION.nzb detected for potential conversion - file not in the correct format, ignoring /home/craig/storage/complete/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION.mkv detected for potential conversion - processing Video codec detected: h264 Audio stream detected: ac3 und [Stream 1] Undefined language detected, defaulting to eng /home/craig/Movies/iTunes/Automatically Add to iTunes.localized/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION.mp4 created /home/craig/storage/complete/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION.mkv deleted /home/craig/storage/complete/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION/the.amazing.race.2301.720p-dimension.nfo detected for potential conversion - file not in the correct format, ignoring /home/craig/storage/complete/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION/the.amazing.race.2301.720p-dimension.srr detected for potential conversion - file not in the correct format, ignoring Loading config from /home/craig/sickbeard_mp4_automator/autoProcess.ini Opening URL: http://localhost:8081/home/postprocess/processEpisode?nzbName=The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION.nzb&quiet=1&dir=%2Fhome%2Fcraig%2Fstorage%2Fcomplete%2FThe.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION Processing folder /home/craig/storage/complete/The.Amazing.Race.S23E01.720p.HDTV.X264-DIMENSION

[mdhiggins]

Ah I know what's happening. You have the output_dir option enabled and the file is getting moved/created before Sickbeard can tag. I'll fix it.

[nctiggy]

Sorry I would help more, still waiting on my laptop to arrive

[mdhiggins]

Try the new update and see if that resolves the tagging issue.

[mdhiggins]

Also I added a 'local_conversion' option with the last commit, see if that works for your iTunes folder issue. I haven't gotten the chance to test it yet. I'm not particularly happy with its implementation it seems kinda messy. I'll see if I can come up with something better [reverted this change, wasn't working and will require a more complicated fix]

Does your setup keep a copy of the mkv and mp4, with sickbeard sending the mkv to its final destination and a copy dropped in iTunes?

[nctiggy]

EDIT: Should mention that the changes on dev did fix the tagging issue from sabnzbd

It is not keeping the original (It is cleaning up) it is just putting the converted file directly into the destination. I think it used to convert in the same location as the source (mkv), tagged it and THEN moved it. Maybe I am wrong though.

Also, Running manual with silent on a folder full of "untagged" mp4's (from earlier ;) ) is not tagging the files. Will need to look into this as well. I can once I get that laptop.

[mdhiggins]

Glad that's fixed.

I'm not having any issues on my end batch-taggin mp4's with the silent option (just tested). Perhaps the guessit feature isn't able to determine their source? What sort of output do you get when you run silent?

[mdhiggins]

I did a big MkvtoMp4 class overhaul, cleaning it up and removing a lot of cluttered legacy code and slimming the whole thing down, making it more modular. I also implemented a copy_to option where multiple directories can be specified (separated with a |) and the final product (converted, tagged, QTFS'd) will be copied to those directories. I think that might be ideal for your setup.

Please give it some thorough bug testing since I rewrote a large portion of the script from scratch.

[nctiggy]

So here is my output from the -silent output on a dir full of MP4's that need tagging, The george martin movie not being guessed is no real surprise, but house hunters should be guessed. Notice the 'movie' reference...Not entirely sure what is going on here...This is with the latest commits today. BTW nice work on the code overhaul, I need understand the copy to piece a bit more, but we can discuss that much later!

craig@nzb:~/sickbeard_mp4_automator$ ./manual.py /home/craig/Movies/iTunes/Home\ Videos/ -silent Processing file /home/craig/Movies/iTunes/Home Videos/George Martin Part 2.mp4 local variable 'movie' referenced before assignment Unable to guess based on filename An unexpected error occured, processing of this file has failed MkvtoMp4 instance has no attribute 'inputfile' Processing file /home/craig/Movies/iTunes/Home Videos/House.Hunters.International.S59E01.720p.hdtv.x264.mp4 local variable 'movie' referenced before assignment Unable to guess based on filename An unexpected error occured, processing of this file has failed MkvtoMp4 instance has no attribute 'inputfile' Processing file /home/craig/Movies/iTunes/Home Videos/WALL·E (2008).mp4 /usr/lib/python2.7/urllib.py:1282: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal return ''.join(map(quoter, s)) u'\xb7' Unable to guess based on filename ^CTraceback (most recent call last): File "./manual.py", line 223, in main() File "./manual.py", line 185, in main walkDir(path, silent) File "./manual.py", line 170, in walkDir processFile(filepath, tagdata) File "./manual.py", line 153, in processFile output = converter.process(inputfile, True) File "/home/craig/sickbeard_mp4_automator/mkvtomp4.py", line 75, in process shutil.copy(inputfile, outputfile) File "/usr/lib/python2.7/shutil.py", line 119, in copy copyfile(src, dst) File "/usr/lib/python2.7/shutil.py", line 84, in copyfile copyfileobj(fsrc, fdst) File "/usr/lib/python2.7/shutil.py", line 49, in copyfileobj buf = fsrc.read(length) KeyboardInterrupt

[mdhiggins]

Alright so I took some time to mimic your file names and track down these errors. You actually had a few different ones. They should be fixed now

The Wall E movie was failing because of the nonunicode character. The George Martin video file was just not able to find a match and due to the way the code was written if no match was found the movie variable was never created, giving you your other errors. The TV show I think was failing because theTVDB was down almost all of yesterday, it was working for me with that exact name with no changes this morning.

Let me know

[nctiggy]

Thanks for the deers. Here is the latest output from Sabnzb on the dev branch

Shark.Tank.S05E03.720p.HDTV.x264-2HD

       Video codec detected: h264

Audio stream detected: ac3 und [Stream 1] Undefined language detected, defaulting to eng /home/craig/storage/complete/Shark.Tank.S05E03.720p.HDTV.x264-2HD/shark.tank.s05e03.720p.hdtv.x264-2hd.mp4 created /home/craig/storage/complete/Shark.Tank.S05E03.720p.HDTV.x264-2HD/shark.tank.s05e03.720p.hdtv.x264-2hd.mkv deleted Loading config from /home/craig/sickbeard_mp4_automator/autoProcess.ini Opening URL: http://localhost:8081/home/postprocess/processEpisode?nzbName=Shark.Tank.S05E03.720p.HDTV.x264-2HD.nzb&quiet=1&dir=%2Fhome%2Fcraig%2Fstorage%2Fcomplete%2FShark.Tank.S05E03.720p.HDTV.x264-2HD Processing folder /home/craig/storage/complete/Shark.Tank.S05E03.720p.HDTV.x264-2HD

Processing /home/craig/storage/complete/Shark.Tank.S05E03.720p.HDTV.x264-2HD/shark.tank.s05e03.720p.hdtv.x264-2hd.mp4 (Shark.Tank.S05E03.720p.HDTV.x264-2HD.nzb)

Found result in history: (100981, 5, [])

Parsed Shark.Tank.S05E03.720p.HDTV.x264-2HD.nzb into Shark Tank - S5E3 - 720p.HDTV.x264 (2HD) [ABD: False]

Checking scene exceptions for a match on Shark Tank

Looking up Shark Tank in the DB

Lookup successful, using tvdb id 100981

Loading show object for tvdb_id 100981

Retrieving episode object for 5x3

The old status had a quality in it, using that: HD TV

SB snatched this episode so I'm marking it as priority

Is ep a priority download: True

There is no existing file so there's no worries about replacing it

This download is marked a priority download so I'm going to replace an existing file if I find one

Found release name Shark.Tank.S05E03.720p.HDTV.x264-2HD

Destination folder for this episode: /home/craig/Movies/zTV/Shark Tank

Moving file from /home/craig/storage/complete/Shark.Tank.S05E03.720p.HDTV.x264-2HD/shark.tank.s05e03.720p.hdtv.x264-2hd.mp4 to /home/craig/Movies/zTV/Shark Tank/shark.tank.s05e03.720p.hdtv.x264-2hd.mp4

Executing command ['/home/craig/sickbeard_mp4_automator/postConversion.py', u'/home/craig/Movies/zTV/Shark Tank/shark.tank.s05e03.720p.hdtv.x264-2hd.mp4', u'/home/craig/storage/complete/Shark.Tank.S05E03.720p.HDTV.x264-2HD/shark.tank.s05e03.720p.hdtv.x264-2hd.mp4', '100981', '5', '3', '2013-10-04']

Absolute path to script: /home/craig/sickbeard_mp4_automator/postConversion.py

Script result: Traceback (most recent call last):

File "/home/craig/sickbeard_mp4_automator/postConversion.py", line 18, in

output = converter.process(inputfile)

File "/home/craig/sickbeard_mp4_automator/mkvtomp4.py", line 84, in process

if self.removeFile(self.inputfile):

AttributeError: MkvtoMp4 instance has no attribute 'inputfile'

Deleting folder /home/craig/storage/complete/Shark.Tank.S05E03.720p.HDTV.x264-2HD

Processing succeeded for /home/craig/storage/complete/Shark.Tank.S05E03.720p.HDTV.x264-2HD/shark.tank.s05e03.720p.hdtv.x264-2hd.mp4

[mdhiggins]

Looks like you're running an older version of the script, that error should be fixed with the most recent.

[nctiggy]

Ok I grabbed that I will report back. On the -silent tag this is what I am seeing in the manual.py

craig@nzb:~/sickbeard_mp4automator$ ./manual.py /home/craig/Movies/iTunes/Home\ Videos/ -silent Processing file /home/craig/Movies/iTunes/Home Videos/..DSStore Skipping /home/craig/Movies/iTunes/Home Videos/..DS_Store - unrecognized filetype Processing file /home/craig/Movies/iTunes/Home Videos/Survivor.S27E03.720p.HDTV.x264-LMAO.mp4 Unable to guess based on filename

for every file...

[mdhiggins]

So the unrecognized filetype error is expected for the random files included in the folder by OS X (.DSStore) and the script is ignoring those as it should.

The Unable to guess based on filename error you're getting is puzzling. On my setup I can rename a random mp4 to the same exact filename as you're reporting (Survivor.S27E03.720p.HDTV.x264-LMAO.mp4) and it works fine. I don't know if maybe one of your files is out of sync or something? Maybe try just clean installing the script except for your autoprocess.ini file. I'm at a loss.

[nctiggy]

When I run the file through the functions in the interactive python shell it returns correctly....

[mdhiggins]

So its able to guess correctly? or you're able to put in the info (tvdbid) manually and it works?

I did mine with the -silent and pointed it to the directory all from the command line (no interactive mode) and it worked fine.

[mdhiggins]

I changed the guessit filename decode from utf8 to ascii. See if that plays nicer with OSX, that might be the problem.

[nctiggy]

I see the problem (I think) is line 173 is sending the entire path of the file. No where downstream does the the path get parsed to become just the filename.

[nctiggy]

filename = os.path.splitext(os.path.basename(filepath))[0]

right before line 173 and then making the change on line 173 may in fact fix the problem

[mdhiggins]

Nah that shouldn't matter, guessit parses out the filename from a full path.

http://guessit.readthedocs.org/en/latest/user/python.html (see the first example they give)

Even better: http://guessit.readthedocs.org/en/latest/dev/internals.html#splitting-into-path-components

[nctiggy]

Found the issue see below: the word Movies is in the path and that key's guessit to think it is a movie. Would passing just the filename to the getinfo method?

craig@nzb:~/sickbeard_mp4_automator$ python Python 2.7.4 (default, Sep 26 2013, 03:20:26) [GCC 4.7.3] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import sys import os import guessit from readSettings import ReadSettings from tvdb_mp4 import Tvdb_mp4 from tmdb_mp4 import tmdb_mp4 from mkvtomp4 import MkvtoMp4 from tvdb_api import tvdb_api from tmdb_api import tmdb from extensions import tmdb_api_key guessit.guess_video_info("/home/craig/Movies/iTunes/Home Videos/Survivor.S27E03.720p.HDTV.x264-LMAO.mp4") {u'mimetype': 'video/mp4', u'videoCodec': u'h264', u'container': u'mp4', u'format': u'HDTV', u'title': u'Survivor S27E03', u'releaseGroup': u'LMAO', u'screenSize': u'720p', u'type': u'movie'}

guessit.guess_video_info("/home/craig/Movies/iTunes/Home_Videos/Survivor.S27E03.720p.HDTV.x264-LMAO.mp4") {u'mimetype': 'video/mp4', u'videoCodec': u'h264', u'container': u'mp4', u'format': u'HDTV', u'title': u'Survivor S27E03', u'releaseGroup': u'LMAO', u'screenSize': u'720p', u'type': u'movie'}

guessit.guess_video_info("/home/craig/Movies/Survivor.S27E03.720p.HDTV.x264-LMAO.mp4") {u'mimetype': 'video/mp4', u'videoCodec': u'h264', u'container': u'mp4', u'format': u'HDTV', u'title': u'Survivor S27E03', u'releaseGroup': u'LMAO', u'screenSize': u'720p', u'type': u'movie'}

guessit.guess_video_info("/home/craig/Survivor.S27E03.720p.HDTV.x264-LMAO.mp4") {u'mimetype': 'video/mp4', u'episodeNumber': 3, u'videoCodec': u'h264', u'container': u'mp4', u'format': u'HDTV', u'series': u'Survivor', u'releaseGroup': u'LMAO', u'screenSize': u'720p', u'season': 27, u'type': u'episode'}

[nctiggy]

Adding filename = os.path.basename(filepath)

before we call the getInfo method and pass filename does resolve this issue.

[mdhiggins]

But do you think its a good idea to remove that functionality? I imagine many people will find it advantageous to have the guessit analyze the directory as well.

[nctiggy]

This could be....hmm I suppose I just need to change my directory structure...

[mdhiggins]

I'm also really struggling right now to resolve some encoding/decoding conflicts for certain special characters (your copy of Wall E with the weird character is still causing issues lol). I guess it would be best to just add an option to use directories when guessing.

[mdhiggins]

Try out the new special character handling I implemented, I would like to know if it works on OS X. I know the old version worked if you ran from command line or had it walk a directory, but would fail if you did a single file via the inputs. Wall E with the little dot between the L and E was my test file lol.

Let me know. I'll have the option in there soon.

[nctiggy]

I can test this later this week. In the mean while have there been any changes that would cause this: Just trying to retag some mp4's

craig@nzb:~/sickbeard_mp4_automator$ ./manual.py /home/craig/Movies/iTunes/Automatically\ Add\ to\ iTunes.localized/3-02\ Uh...\ Oh...\ Ah…\ (HD).mp4 -tv 247897 3 2 Processing Homeland Season 3 Episode 2 - Uh... Oh... Ah… Error moving file to output directory /home/craig/Movies/iTunes/Automatically Add to iTunes.localized/3-02 Uh... Oh... Ah… (HD).mp4 and /home/craig/Movies/iTunes/Automatically Add to iTunes.localized/3-02 Uh... Oh... Ah… (HD).mp4 are the same file Tagging file :/home/craig/Movies/iTunes/Automatically Add to iTunes.localized/3-02 Uh... Oh... Ah… (HD).mp4 Traceback (most recent call last): File "./manual.py", line 235, in main() File "./manual.py", line 216, in main processFile(path, tagdata) File "./manual.py", line 169, in processFile tagmp4.writeTags(output['output']) File "/home/craig/sickbeard_mp4_automator/tvdb_mp4.py", line 54, in writeTags MP4(mp4Path).delete() File "/home/craig/sickbeard_mp4_automator/mutagen/init.py", line 75, in init self.load(filename, _args, _kwargs) File "/home/craig/sickbeard_mp4_automator/mutagen/mp4.py", line 710, in load try: self.tags = self.MP4Tags(atoms, fileobj) File "/home/craig/sickbeard_mp4_automator/mutagen/_util.py", line 105, in init super(DictProxy, self).init(_args, _kwargs) File "/home/craig/sickbeard_mp4_automator/mutagen/init.py", line 39, in init self.load(_args, _kwargs) File "/home/craig/sickbeard_mp4_automator/mutagen/mp4.py", line 251, in load info[0](self, atom, data, info[2:]) File "/home/craig/sickbeard_mp4_automator/mutagen/mp4.py", line 549, in parse_uint32 value in self.__parse_data(atom, data)] File "/home/craig/sickbeard_mp4_automator/mutagen/mp4.py", line 419, in parse_data length, name, flags = struct.unpack(">I4sI", data[pos:pos+12]) struct.error: unpack requires a string argument of length 12

[mdhiggins]

This might have something to do with the fact that the title of that episode is using a weird unicode character for the last ... in the title of that episode. I was able to tag the file without issue, except when outputting the line Tagging file :/home/craig/Movies/iTunes/Automatically Add to iTunes.localized/3-02 Uh... Oh... Ah… (HD).mp4 in Windows caused an error. Latest commit fixed that, but it seems on OS X this didn't create an issue since OS X terminal uses unicode by default.

I went ahead and changed the title in theTVDB to remove the unicode character. See if it works. If its still giving you an error, I would remux the mp4 into a fresh mp4. A lot of times mp4 files downloaded from the internet don't have properly setup mp4 moov atoms and when it tries to write the tags it runs into trouble. Let me know what works so I can chase down the bugs further.

[mdhiggins]

Got my Hackintosh build up and running and did some OS X testing today. Check out the latest commits

[mdhiggins]

Closing this since the dev branch is gone, if you run into problems please open a new issue. Thanks for all the feedback