search

mdhiggins / sickbeard_mp4_automator

Automatically convert video files to a standardized format with metadata tagging to create a beautiful and uniform media library
MIT License
1.53k stars 201 forks source link

SMA not working after being triggered on File Rename #1334

Closed dramirvf closed 4 years ago

dramirvf commented 4 years ago

I have temporarily set the MP4 Automator scrip to be triggered on file rename in Radarr. I did this to trigger tagging for movies which are currently listed in Radarr, but MP4 Automator has not processed them so far. However, after mass renaming in Radarr, the script does not work as expected and I see the following logs for every file which has been renamed:

2020-10-03 13:08:42 - RadarrPostProcess - INFO - Radarr extra script post processing started. 2020-10-03 13:08:42 - resources.readsettings - INFO - /usr/bin/python3 2020-10-03 13:08:42 - resources.readsettings - INFO - Loading config file /usr/local/sma/config/autoProcess.ini. 2020-10-03 13:08:42 - resources.readsettings - WARNING - Force-convert is true, so process-same-extensions is being overridden to true as well 2020-10-03 13:08:42 - RadarrPostProcess - ERROR - Error renaming inputfile Traceback (most recent call last): File "/usr/local/sma/postRadarr.py", line 104, in inputfile = renameFile(inputfile, log) File "/usr/local/sma/postRadarr.py", line 53, in renameFile filename, fileext = os.path.splitext(inputfile) File "/usr/lib/python3.8/posixpath.py", line 118, in splitext p = os.fspath(p) TypeError: expected str, bytes or os.PathLike object, not NoneType 2020-10-03 13:08:42 - resources.mediaprocessor - ERROR - isValidSource unexpectedly threw an exception, returning None Traceback (most recent call last): File "/usr/local/sma/resources/mediaprocessor.py", line 262, in isValidSource extension = self.parseFile(inputfile)[2] File "/usr/local/sma/resources/mediaprocessor.py", line 1554, in parseFile path = os.path.abspath(path) File "/usr/lib/python3.8/posixpath.py", line 374, in abspath path = os.fspath(path) TypeError: expected str, bytes or os.PathLike object, not NoneType 2020-10-03 13:08:42 - resources.mediaprocessor - INFO - File None is not valid 2020-10-03 13:08:42 - RadarrPostProcess - INFO - Processing returned False. 2020-10-03 13:08:42 - RadarrPostProcess - ERROR - Error processing file Traceback (most recent call last): File "/usr/local/sma/postRadarr.py", line 170, in sys.exit(1) SystemExit: 1

Is there any additional settings to resolve this issue?

mdhiggins commented 4 years ago

Script cannot be launched from rename. Radarr only provides a very small amount of data when that event is called. Use the manual import feature to reprocess already downloaded files.

Sent from my iPhone

On Oct 3, 2020, at 6:32 AM, dramirvf notifications@github.com wrote:

 I have temporarily set the MP4 Automator scrip to be triggered on file rename in Radarr. I did this to trigger tagging for movies which are currently listed in Radarr, but MP4 Automator has not processed them so far. However, after mass renaming in Radarr, the script does not work as expected and I see the following logs for every file which has been renamed:

2020-10-03 13:08:42 - RadarrPostProcess - INFO - Radarr extra script post processing started. 2020-10-03 13:08:42 - resources.readsettings - INFO - /usr/bin/python3 2020-10-03 13:08:42 - resources.readsettings - INFO - Loading config file /usr/local/sma/config/autoProcess.ini. 2020-10-03 13:08:42 - resources.readsettings - WARNING - Force-convert is true, so process-same-extensions is being overridden to true as well 2020-10-03 13:08:42 - RadarrPostProcess - ERROR - Error renaming inputfile Traceback (most recent call last): File "/usr/local/sma/postRadarr.py", line 104, in inputfile = renameFile(inputfile, log) File "/usr/local/sma/postRadarr.py", line 53, in renameFile filename, fileext = os.path.splitext(inputfile) File "/usr/lib/python3.8/posixpath.py", line 118, in splitext p = os.fspath(p) TypeError: expected str, bytes or os.PathLike object, not NoneType 2020-10-03 13:08:42 - resources.mediaprocessor - ERROR - isValidSource unexpectedly threw an exception, returning None Traceback (most recent call last): File "/usr/local/sma/resources/mediaprocessor.py", line 262, in isValidSource extension = self.parseFile(inputfile)[2] File "/usr/local/sma/resources/mediaprocessor.py", line 1554, in parseFile path = os.path.abspath(path) File "/usr/lib/python3.8/posixpath.py", line 374, in abspath path = os.fspath(path) TypeError: expected str, bytes or os.PathLike object, not NoneType 2020-10-03 13:08:42 - resources.mediaprocessor - INFO - File None is not valid 2020-10-03 13:08:42 - RadarrPostProcess - INFO - Processing returned False. 2020-10-03 13:08:42 - RadarrPostProcess - ERROR - Error processing file Traceback (most recent call last): File "/usr/local/sma/postRadarr.py", line 170, in sys.exit(1) SystemExit: 1

Is there any additional settings to resolve this issue?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

mdhiggins commented 4 years ago

1333

Someone just asked about this

dramirvf commented 4 years ago

The problem with importing again is that during import, Radarr will match the movies from scratch. So many of the matches will be incorrect.

Do you have any recommendation on how to manual import again so we make sure the exact matches (which is already corrected before manual import) would be sent to SMA Automator?

mdhiggins commented 4 years ago

It will still match them against what's in your database and if you configure your naming conventions so there's no ambiguity you should be fine. Just have radarr put the IMDB ID in the name before the bulk manual import

dramirvf commented 4 years ago

IMDB ID or TMDB ID?

mdhiggins commented 4 years ago

I don't believe radarr can do tmdb id in the name automatically

mdhiggins commented 4 years ago

Bulk rename everything to include ID, move files you want to process to a temp directory, manually import, rename back to however you like

dramirvf commented 4 years ago

It does. Since Radarr relies mostly on TMDB ID so I included it in the filename through bulk rename. I'm doing manual import now. If it works, this solution is very handy. Because many people want to start using SMA Automator after several years of using Radarr. If the matches for already downloaded movies are incorrect, the whole job during several years of working with Radarr will ruin in seconds.

I'll update the results.

dramirvf commented 4 years ago

Bulk rename everything to include ID, move files you want to process to a temp directory, manually import, rename back to however you like

I am doing exactly the same.

mdhiggins commented 4 years ago

Yeah I've used that workflow for migration before and its always been perfect, I add the ID and release quality to the name before manual import and the match should be fine

dramirvf commented 4 years ago

I moved 136 folders to a temporary folder and I did a manual import in Radarr. All 136 movies have been imported to Radarr without any issues. All 136 have the correct TMDB ID assigned to them and there is no issues in the Radarr side.

After checking the files, however, I noticed there are some files which have no tags on them. I checked the SMA logs and I noticed that according to the 4 available log files, there are 17 movies that have not been tagged by SMA with a similar error message. The actual number should be more, but only 4 log files are available and I don't know why SMA has discarded the earlier logs.

I checked those 17 files and all of them have valid TMDB ID and there is no issue associated with them in Radarr. All error messages for these 17 files in SMA logs are exactly the same. They all indicate that "SMA is Unable to resolve a valid TMDBID for the file and therefore SMA is Unable to tag file". Below is the full log for one of the movies:

2020-10-03 18:59:27 - resources.readsettings - INFO - /usr/bin/python3 2020-10-03 18:59:27 - resources.readsettings - INFO - Loading config file /usr/local/sma/config/autoProcess.ini. 2020-10-03 18:59:27 - resources.readsettings - WARNING - Force-convert is true, so process-same-extensions is being overridden to true as well 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Processing /movies/Movies - TMP/Two Patches of Clouds (2015)/Two Patches of Clouds (2015) HDTV-720p [8-Bit][x264][AAC 1.0][TMDB-626871].rnm.mp4. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Input Data 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - { "format": "mov,mp4,m4a,3gp,3g2,mj2", "format-fullname": "QuickTime / MOV", "video": { "index": 0, "codec": "h264", "bitrate": 1330838, "pix_fmt": "yuv420p", "profile": "high", "fps": 25.0, "dimensions": "1274x720", "level": 3.1, "field_order": "unknown" }, "audio": [ { "index": 1, "codec": "aac", "bitrate": 64000, "channels": 1, "samplerate": 44100, "language": "fas", "disposition": "+default-dub-original-comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired" } ], "subtitle": [], "attachment": [] } 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Reading video stream. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Video codec detected: h264. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Pix Fmt: yuv420p. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Profile: high. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Video codec parameters None. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Creating copy video stream from source stream 0. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Reading audio streams. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Audio detected for stream 1 - aac fas 1 channel. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Creating ac3 audio stream from source stream 1. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Default audio stream set to fas ac3 1 channel stream [default-more-channels: True]. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Reading subtitle streams. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Scanned for external subtitles and found 0 results in your approved languages. 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Output Data 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - { "source": [ "/movies/Movies - TMP/Two Patches of Clouds (2015)/Two Patches of Clouds (2015) HDTV-720p [8-Bit][x264][AAC 1.0][TMDB-626871].rnm.mp4" ], "format": "mp4", "video": { "codec": "copy", "map": 0, "bitrate": 1270.568, "crf": -1, "maxrate": null, "bufsize": null, "level": 0.0, "profile": null, "preset": null, "pix_fmt": null, "field_order": "unknown", "width": null, "filter": null, "params": null, "title": "HD", "debug": "video" }, "audio": [ { "map": 1, "codec": "ac3", "channels": 1, "bitrate": 64.0, "filter": null, "samplerate": null, "language": "fas", "disposition": "+default-dub-original-comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired", "bsf": null, "title": "Mono", "debug": "universal-audio" } ], "subtitle": [], "attachment": [] } 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Preopts 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - [] 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Postopts 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - [ "-threads", "0", "-metadata:g", "encoding_tool=SMA" ] 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Subtitle Extracts 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - [] 2020-10-03 18:59:27 - resources.mediaprocessor - INFO - Starting conversion. 2020-10-03 18:59:28 - resources.mediaprocessor - INFO - FFmpeg command: 2020-10-03 18:59:28 - resources.mediaprocessor - INFO - ====================== 2020-10-03 18:59:28 - resources.mediaprocessor - INFO - /usr/local/bin/ffmpeg -i /movies/Movies - TMP/Two Patches of Clouds (2015)/Two Patches of Clouds (2015) HDTV-720p [8-Bit][x264][AAC 1.0][TMDB-626871].rnm.mp4.original -vcodec copy -map 0:0 -metadata:s:v title=HD -c:a:0 ac3 -map 0:1 -ac:a:0 1 -b:a:0 64k -metadata:s:a:0 BPS=64000 -metadata:s:a:0 BPS-eng=64000 -metadata:s:a:0 title=Mono -metadata:s:a:0 language=fas -disposition:a:0 +default-dub-original-comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired-captions -f mp4 -threads 0 -metadata:g encoding_tool=SMA -y /movies/Movies - TMP/Two Patches of Clouds (2015)/Two Patches of Clouds (2015) HDTV-720p [8-Bit][x264][AAC 1.0][TMDB-626871].rnm.mp4 2020-10-03 18:59:28 - resources.mediaprocessor - INFO - ====================== 2020-10-03 18:59:55 - resources.mediaprocessor - INFO - /movies/Movies - TMP/Two Patches of Clouds (2015)/Two Patches of Clouds (2015) HDTV-720p [8-Bit][x264][AAC 1.0][TMDB-626871].rnm.mp4 created. 2020-10-03 18:59:55 - resources.metadata - ERROR - Unable to resolve a valid TMDBID. 2020-10-03 18:59:55 - resources.mediaprocessor - ERROR - Unable to tag file Traceback (most recent call last): File "/usr/local/sma/resources/mediaprocessor.py", line 49, in fullprocess tag = Metadata(mediatype, tvdbid=tvdbid, tmdbid=tmdbid, imdbid=imdbid, season=season, episode=episode, original=original, language=language) File "/usr/local/sma/resources/metadata.py", line 60, in init raise TMDBIDError resources.metadata.TMDBIDError 2020-10-03 18:59:55 - resources.mediaprocessor - INFO - Relocating MOOV atom to start of file. 2020-10-03 18:59:55 - qtfaststart - INFO - Removing free atom at 32 (8 bytes) 2020-10-03 18:59:55 - qtfaststart - INFO - Patching stco with 130210 entries 2020-10-03 18:59:55 - qtfaststart - INFO - Patching stco with 130210 entries 2020-10-03 18:59:55 - qtfaststart - INFO - Writing output... 2020-10-03 18:59:59 - RadarrPostProcess - INFO - Radarr response Rescan command: ID 131042 queued. 2020-10-03 18:59:59 - RadarrPostProcess - INFO - Waiting rescan to complete 2020-10-03 19:00:09 - RadarrPostProcess - INFO - Final state: completed. 2020-10-03 19:00:09 - RadarrPostProcess - INFO - Rescan command completed 2020-10-03 19:00:09 - RadarrPostProcess - INFO - Requesting updated information from Radarr for movie ID 5807. 2020-10-03 19:00:19 - RadarrPostProcess - INFO - Radarr monitoring information updated for movie Two Patches of Clouds. 2020-10-03 19:00:19 - RadarrPostProcess - INFO - Radarr response Rename command: ID 131044 queued. 2020-10-03 19:00:24 - RadarrPostProcess - INFO - Radarr extra script post processing started.

Any suggestions?

dramirvf commented 4 years ago

I checked all the files with MediaInfo and I confirm that 51 files out of 136 files are not tagged. Meanwhile, all of the 136 files have been assigned the correct TMDB ID in Radarr and matched with not even a single issue.

So it looks like that during manual import, SMA does not work according to the TMDB ID which is being passed through Radarr or the TMDB ID which is included in the filename. It clearly tries to match the files by itself, which has failed with a very high rate here (37.5% of imported files).

dramirvf commented 4 years ago

I believe it is very wise to include a feature in the python script, so it give a preference to the presence of TMDB ID or IMDB ID in the filename and in such scenarios it could bypass the guessit dependency to match the files by itself.

mdhiggins commented 4 years ago

Your conclusion is incorrect, it does not guess but just uses the IMDB ID that is passed from Radarr (not TMDB ID)

This is mostly historic as Radarr did not previously pass the TMDB ID at the time postRadarr.py was developed (https://github.com/Radarr/Radarr/wiki/Custom-Post-Processing-Scripts/_compare/c15618f77b93a751695a416eea49c5996e1e1c59...07a2dec3c5775309615dbdecb39c121a700aab02)

That IMDBID is then resolved to a TMDBID using the TMDB api

I'm guessing the failures may be from movies where the TMDB api doesn't know the IMDB ID and thus can't resolve it; hence the "ERROR - Unable to resolve a valid TMDBID."

That being said, the system was designed to be a hierarchy and accept any ID and attempt to resolve it to a TMDB ID, supporting TMDB, TVDB, and IMDB, with TMDB being the highest in the chain

I had been unaware Radarr started providing TMDB as an environment variable, but since it does its just a simple update to pull that variable and pass it to the tagger

But no guessing is taking place, just ID to ID resolution

00be9e0f33b3958305b63f9be1a242eeb67d76de

That should fix it, please test one of the failed files and see

mdhiggins commented 4 years ago

The only file that touches the guessit library is manual.py

mdhiggins commented 4 years ago 
{
   "id":603,
   "imdb_id":"tt0133093",
   "title":"The Matrix",
}
{
   "id":626871,
   "imdb_id":"None",
   "title":"Two Patches of Clouds",
}

Checked the data in TMDB and was correct, no IMDB ID for that movie that failed

dramirvf commented 4 years ago

You are right. My conclusion was totally incorrect. Now that the TMDB ID is passed to SMA, the problem is solved. I checked the updated script on one of the files and it was tagged properly.

Btw is there any help or guide available on how to configure logging.ini? I want to keep more logs but I could not find any guides like what you have provided for the configuration of autoProcess.ini.

mdhiggins commented 4 years ago

Glad it's working and that fixed things

Logging.ini is just a standard python logging configuration file you can read about it in the python docs

https://docs.python.org/3/library/logging.config.html