clinton-hall / nzbToMedia

Provides NZB and Torrent postprocessing To CouchPotatoServer, SickBeard/SickRage, HeadPhones, Mylar and Gamez
GNU General Public License v3.0
673 stars 176 forks source link

NZBtoMedia via SABnzbd fails to post-process Radarr download #1987

Closed DerekAlldritt closed 7 months ago

DerekAlldritt commented 9 months ago

I've recently got a problem processing movies with NZBtoMedia and can't figure out what has changed. Ironically, we run 2 servers with (supposedly) identical setups and one processes properly, and one has the following issue.

Can be found originally posted at https://forum.qnap.com/viewtopic.php?p=859689#p859689

Basics are Radarr to SAB, SAB downloads and tries to post-process and barfs with the following error.

processing Exit(1): json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[code]/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/libs/common/beets/mediafile.py:19: UserWarning: beets.mediafile is deprecated; use mediafile instead
warnings.warn("beets.mediafile is deprecated; use mediafile instead")
[09:13:03] [INFO]::MAIN: Loading config from [/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/autoProcessMedia.cfg]
[09:13:03] [INFO]::MAIN: Python v3.11 will reach end of life in 1317 days.
[09:13:03] [INFO]::MAIN: Checking database structure...
[09:13:03] [INFO]::MAIN: Checking if git needs an update
[09:13:04] [INFO]::MAIN: No update needed
[09:13:04] [INFO]::MAIN: nzbToMedia Version:558970c212343bdc54a26f61f8cb317d27a26c3e Branch:master (Linux 5.10.60-qnap)
[09:13:04] [INFO]::MAIN: #########################################################
[09:13:04] [INFO]::MAIN: ## ..::[nzbToMedia.py]::.. ##
[09:13:04] [INFO]::MAIN: #########################################################
[09:13:04] [INFO]::MAIN: Script triggered from SABnzbd 4.2.2.
[09:13:04] [INFO]::MAIN: Auto-detected SECTION:Radarr
[09:13:04] [INFO]::MAIN: Calling Radarr:movie to post-process:Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA
[09:13:04] [INFO]::MAIN: Attemping imdbID lookup for Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA
[09:13:04] [INFO]::MAIN: Searching folder and file names for imdbID ...
[09:13:04] [INFO]::MAIN: Searching IMDB for imdbID ...
[09:13:04] [INFO]::MAIN: Unable to determine imdbID: No api key provided for omdbapi.com.
[09:13:04] [INFO]::TRANSCODER: Checking [Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA.mkv] for corruption, please stand by ...
[09:13:04] [INFO]::TRANSCODER: SUCCESS: [Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA.mkv] has no corruption.
[09:13:04] [POSTPROCESS]::RADARR: Starting DownloadedMoviesScan scan for Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA
Changing to directory: /share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache

-- Cleaning bytecode --
b'Removing __pycache__/\n'
b'Removing __pycache__/\n'

-- Cleaning folders: ['libs', 'core'] --
No folders to clean

Returning to directory:  /share/CACHEDEV1_DATA/.qpkg/SABnzbd/repo-cache

-- Cleanup finished --

Traceback (most recent call last):
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/nzbToRadarr.py", line 272, in <module>
result = nzbToMedia.main(sys.argv, section)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/nzbToMedia.py", line 763, in main
result = sab.process_script()
^^^^^^^^^^^^^^^^^^^^
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/core/processor/sab.py", line 13, in process_script
return nzb.process(
^^^^^^^^^^^^
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/core/processor/nzb.py", line 125, in process
result = movies.process(section_name, input_directory, input_name, status, client_agent, download_id, input_category, failure_link)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/core/auto_process/movies.py", line 247, in process
result = r.json()
^^^^^^^^
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/libs/common/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/lib/python3.11/json/__init__.py", line 346, in loads
File "/opt/lib/python3.11/json/decoder.py", line 337, in decode
File "/opt/lib/python3.11/json/decoder.py", line 355, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

and here is the properly processed one on the other server

/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/libs/common/beets/mediafile.py:19: UserWarning: beets.mediafile is deprecated; use mediafile instead
warnings.warn("beets.mediafile is deprecated; use mediafile instead")
[09:23:17] [INFO]::MAIN: Loading config from [/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/autoProcessMedia.cfg]
[09:23:17] [INFO]::MAIN: Python v3.11 will reach end of life in 1317 days.
[09:23:17] [INFO]::MAIN: Checking database structure...
[09:23:17] [INFO]::MAIN: Checking if git needs an update
[09:23:18] [INFO]::MAIN: No update needed
[09:23:18] [INFO]::MAIN: nzbToMedia Version:558970c212343bdc54a26f61f8cb317d27a26c3e Branch:master (Linux 5.10.60-qnap)
[09:23:18] [INFO]::MAIN: #########################################################
[09:23:18] [INFO]::MAIN: ## ..::[nzbToMedia.py]::.. ##
[09:23:18] [INFO]::MAIN: #########################################################
[09:23:18] [INFO]::MAIN: Script triggered from SABnzbd 4.2.2.
[09:23:18] [INFO]::MAIN: Auto-detected SECTION:Radarr
[09:23:18] [INFO]::MAIN: Calling Radarr:movie to post-process:Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA
[09:23:18] [INFO]::MAIN: Attemping imdbID lookup for Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA
[09:23:18] [INFO]::MAIN: Searching folder and file names for imdbID ...
[09:23:18] [INFO]::MAIN: Searching IMDB for imdbID ...
[09:23:18] [INFO]::MAIN: Unable to determine imdbID: No api key provided for omdbapi.com.
[09:23:18] [INFO]::TRANSCODER: Checking [Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA.mkv] for corruption, please stand by ...
[09:23:18] [INFO]::TRANSCODER: SUCCESS: [Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA.mkv] has no corruption.
[09:23:18] [POSTPROCESS]::RADARR: Starting DownloadedMoviesScan scan for Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA
[09:23:18] [POSTPROCESS]::RADARR: Checking for status change, please stand by ...
[09:24:19] [POSTPROCESS]::RADARR: Checking for status change, please stand by ...
[09:24:19] [INFO]::MAIN: The /share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/nzbToRadarr.py script completed successfully.
Changing to directory: /share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache

-- Cleaning bytecode --
b'Removing __pycache__/\n'
b'Removing __pycache__/\n'

-- Cleaning folders: ['libs', 'core'] --
No folders to clean

Returning to directory:  /share/CACHEDEV1_DATA/.qpkg/SABnzbd/repo-cache

-- Cleanup finished --

Radarr: Successfully post-processed Leave.the.World.Behind.2023.1080p.10bit.WEBRip.6CH.x265.HEVC-PSA!
clinton-hall commented 8 months ago

This should be fixed in nightly branch. Are you able to switch to nightly and test?

OneCDOnly commented 8 months ago

@DerekAlldritt, to do this in the nzbtoMedia QPKG requires a bit of editing (this is no-longer necessary for most of my QPKGs but nzbToMedia is an exception as it works a little differently so it's not been updated yet).

Here's a shell command to perform the edit automatically:

sed -i 's|SOURCE_GIT_BRANCH=master|SOURCE_GIT_BRANCH=nightly|' $(getcfg nzbToMedia Install_Path -f /etc/config/qpkg.conf)/nzbtomedia.sh 

... then restart the QPKG:

/etc/init.d/nzbtomedia.sh restart
clinton-hall commented 8 months ago

thanks @OneCDOnly I'm hoping to merge to master soon. But as this officially kills off python2.7 and 3.7 support I want to get a bit of testing done before I merge.

OneCDOnly commented 8 months ago

BTW: @clinton-hall, while I think of it: should we be running nzbToMedia inside the same Python virtualenv as SABnzbd?

I don't presently do this in the QPKG, as I've given nzbToMedia its own virtualenv, but I'm not sure that's the right way to do-it.

Any thoughts on this please? Your advice is always appreciated. :)

clinton-hall commented 8 months ago

This could get very philosophical... I don't think there is a "right" way.

Basically I can see arguments either way. In terms of security it is probably better to have nzbToMedia in its own venv and only interacting via api and access to shared folders.

In terms of simplicity, with the script being called by SABnzbd, having it inside of the SABnzbd venv is probably cleaner.

I'm not sure if either option will make it easier for users to maintain etc?

OneCDOnly commented 8 months ago

If nzbToMedia is in its own virtualenv and is called by SABnzbd, which Python modules will be accessible to nzbToMedia? I've tried solving that beets.mediafile error a few times, but no-luck so-far as it seems calling the post-processing script from SABnzbd doesn't load the nzbToMedia Python virtualenv.

clinton-hall commented 8 months ago

Ah.... I hadn't thought about that. In theory, to call nzbToMedia within its own venv you would probably need a shell/bash script that loads the nzbToMedia venv and then call nzbToMedia (passing all of the input variables through). Generally it is going to be called within the SABnzbd environment unless another environment is specifically loaded.

OneCDOnly commented 8 months ago

Ah, that makes sense. Thank you. 👍🏽

I'm not sure how to keep nzbToMedia as a distinct QPKG if the best-way to run it is inside the SABnzbd virtualenv. Something for me to figure-out.

Cheers mate! :)

clinton-hall commented 8 months ago

In theory, if nzbToMedia has it's own venv, you would set SABnzbd to call a shell/bash script that does something like this.

source <path to nzbToMedia env> activate
python <path to nzbtomedia.py> "$@"

I can't recall if you need to quote the args (as I have here) or not.

You may also want to specifically capture the return code and pass that back. Or even pass through the entire output etc. e.g.

source <path to nzbToMedia env> activate
python <path to nzbtomedia.py> "$@" 2>&1
echo $?
OneCDOnly commented 8 months ago

Good idea.

I think this would be better, as it allows SABnzbd and nzbToMedia to remain separate. I'll create a bash script as you've suggested to MITM the two. 🤓

DerekAlldritt commented 8 months ago

@DerekAlldritt, to do this in the nzbtoMedia QPKG requires a bit of editing (this is no-longer necessary for most of my QPKGs but nzbToMedia is an exception as it works a little differently so it's not been updated yet).

Here's a shell command to perform the edit automatically:

sed -i 's|SOURCE_GIT_BRANCH=master|SOURCE_GIT_BRANCH=nightly|' $(getcfg nzbToMedia Install_Path -f /etc/config/qpkg.conf)/nzbtomedia.sh 

... then restart the QPKG:

/etc/init.d/nzbtomedia.sh restart

As per request, this was run. Error has now changed.

/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/libs/common/beets/mediafile.py:19: UserWarning: beets.mediafile is deprecated; use mediafile instead
warnings.warn("beets.mediafile is deprecated; use mediafile instead")
[08:22:50] [INFO]::MAIN: Loading config from [/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/autoProcessMedia.cfg]
[08:22:50] [INFO]::MAIN: Python v3.11 will reach end of life in 1311 days.
[08:22:50] [INFO]::MAIN: Checking database structure...
[08:22:50] [INFO]::MAIN: Database upgrade required: Initial Schema
[08:22:50] [INFO]::MAIN: Checking if git needs an update
[08:22:50] [INFO]::MAIN: No update needed
[08:22:50] [INFO]::MAIN: nzbToMedia Version:f98d6fff65350dfb884efdfd77609e7aae4c9fd5 Branch:nightly (Linux 5.10.60-qnap)
[08:22:50] [INFO]::MAIN: #########################################################
[08:22:50] [INFO]::MAIN: ## ..::[nzbToMedia.py]::.. ##
[08:22:50] [INFO]::MAIN: #########################################################
[08:22:50] [INFO]::MAIN: Script triggered from SABnzbd 4.2.2.
[08:22:50] [INFO]::MAIN: Auto-detected SECTION:Radarr
[08:22:50] [INFO]::MAIN: Calling Radarr:movie to post-process:Higuita.The.Way.of.the.Scorpion.2023.720p.WEB.h264-EDITH
[08:22:50] [INFO]::MAIN: Attemping imdbID lookup for Higuita.The.Way.of.the.Scorpion.2023.720p.WEB.h264-EDITH
[08:22:50] [INFO]::MAIN: Searching folder and file names for imdbID ...
[08:22:50] [INFO]::MAIN: Searching IMDB for imdbID ...
[08:22:50] [INFO]::MAIN: Unable to determine imdbID: No api key provided for omdbapi.com.
Changing to directory: /share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache

-- Cleaning bytecode --
b'Removing __pycache__/\n'
b'Removing __pycache__/\n'

-- Cleaning folders: ['libs', 'core'] --
No folders to clean

Returning to directory:  /share/CACHEDEV1_DATA/.qpkg/SABnzbd/repo-cache

-- Cleanup finished --

Traceback (most recent call last):
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/nzbToRadarr.py", line 272, in <module>
result = nzbToMedia.main(sys.argv, section)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/nzbToMedia.py", line 763, in main
result = sab.process_script()
^^^^^^^^^^^^^^^^^^^^
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/core/processor/sab.py", line 13, in process_script
return nzb.process(
^^^^^^^^^^^^
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/core/processor/nzb.py", line 125, in process
result = movies.process(section_name, input_directory, input_name, status, client_agent, download_id, input_category, failure_link)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/share/CACHEDEV1_DATA/.qpkg/nzbToMedia/repo-cache/core/auto_process/movies.py", line 69, in process
imdbid, dir_name = find_imdbid(dir_name, input_name, omdbapikey)
^^^^^^^^^^^^^^^^
TypeError: cannot unpack non-iterable NoneType object
clinton-hall commented 8 months ago

Sorry for this, and thanks for the log. When an improvement was made for the imdbid identification, this added extra dir_name return. But this wasn't added when no imdbid was found!

I have fixed this now via https://github.com/clinton-hall/nzbToMedia/commit/b7d6ad8c0752832150299dbd4e38c68f59125eef if you run again, hopefully this will work.

OneCDOnly commented 8 months ago

@DerekAlldritt, please restart the nzbToMedia QPKG to refresh your current git clone. :)