search

clinton-hall / nzbToMedia

Provides NZB and Torrent postprocessing To CouchPotatoServer, SickBeard/SickRage, HeadPhones, Mylar and Gamez
GNU General Public License v3.0
670 stars 176 forks source link

Medusa API requires 'path' instead of 'proc_dir' #1383

Closed rigerc closed 5 years ago

rigerc commented 5 years ago

After a whole day trying to figure out why Medusa won't process anything I feed it using nzbToMedia, I noticed that 'proc_dir' gets passed to the API, while Medusa expects 'path' (per the API documentation that's apparently only available inside the Medusa install, can't find it on the web).

After changing it it works flawlessly. The offending issue is at https://github.com/clinton-hall/nzbToMedia/blob/master/core/__init__.py#L76

I'm not a programmer and don't know if I'm right, just that it works. Maybe someone could take a look at it.

Fwiw, I'm not using autofork since it crashed the script, so I set it to 'medusa'. If I read this correctly Medusa shouldn't work with API only set but it does for me atm.

clinton-hall commented 5 years ago

Ok, I can't apply that change right now, as that would break the non-api interface, and would stop the auto fork for working with Medusa.

So what needs to happen here is to confirm all of the optional parameters, and the format of the api base page in Medusa to build a fully functional auto-form that works with Medusa api.

rigerc commented 5 years ago

OK, I ran the script without API key, autofork set to auto to get the param's and everything seems to work. This is the relevant output:

[14:07:43] [INFO]::MAIN: Attempting to auto-detect tv fork

[14:07:45] [INFO]::MAIN: SickBeard:tv fork auto-detection found custom params {'force': None, 'failed': None, 'process_m ethod': None, 'proc_dir': None, 'type': None, 'delete_on': None}

[14:07:45] [INFO]::MAIN: SickBeard:tv fork set to custom

[14:07:45] [POSTPROCESS]::SICKBEARD: SUCCESS: The download succeeded, sending a post-process request

[14:07:45] [DEBUG]::SICKBEARD: Opening URL: http://localhost:8083/home/postprocess/processEpisode with params: {'nzbName ': 'filename.mkv', 'quiet': 1, 'failed': 0, 'process_method': 'move', 'proc_di r': '/volume1/downloads/process/tv/filename.mkv', 'type': 'manual'}

Which I find odd because it's using proc_dir for the custom fork and it's working fine in Medusa.

clinton-hall commented 5 years ago

Can you test with the api key again, and fork = auto? I have just made some changes on latest nightly build.

medariox commented 5 years ago

proc_dir is actually correct, see: https://github.com/pymedusa/Medusa/blob/develop/medusa/server/web/home/post_process.py#L25-L26

clinton-hall commented 5 years ago

yes, prod_dir is correct for the web ui (which required user and pass authentication). If you want to use api, then this is available to test in nightly branch. I probably still need to make some adjustments as this has been build based on the SickRage and SickChill versions of the api.

a10kiloham commented 5 years ago

Hi @clinton-hall can you confirm this is all pushed to the main branch now? For Medusa what's the preferred path - include API key and type = auto? I'm still getting inconsistent results.

clinton-hall commented 5 years ago

@a10kiloham Yes, this has been merged into master. Putting in the api and setting fork = auto is the recommended settings.

What "inconsistent results" are you seeing. Can you describe what is/isn't happening and provide logs?

a10kiloham commented 5 years ago

So I'm constantly getting Medusa errors

2018-12-07 08:35:11 WARNING  TORNADO :: [54da0c7] Please consider manually move torrent to seed folder as there is no info hash in snatch history:  /Media/Videos/tv/TSHOWNAME720p.HEVC.x265-MeGusta-xpost/SHOWNAME.720p.HEVC.x265-MeGusta-xpost.mkv

nzbToMedia logs show the following:

2018-12-07 08:34:55 DEBUG   ::MAIN: Options passed into nzbToMedia: ['/nzbToMedia/nzbToSickBeard.py', '/Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost', 'SHOWNAME.720p.HEVC.x265-MeGusta-xpost.nzb', 'SHOWNAME.720p.HEVC.x265-MeGusta-xpost', '', 'tv', 'alt.binaries.misc', '0', '']
2018-12-07 08:34:55 INFO    ::MAIN: Script triggered from SABnzbd 0.7.17+
2018-12-07 08:34:55 DEBUG   ::MAIN: Searching for nzoid from SAbnzbd ...
2018-12-07 08:34:55 DEBUG   ::MAIN: Found nzoid: SABnzbd_nzo_mqWyJu
2018-12-07 08:34:55 DEBUG   ::MAIN: Adding NZB download info for directory /Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost to database
2018-12-07 08:34:55 DB      ::MAIN: nzbtomedia.db: UPDATE downloads SET status = ?, client_agent = ?, last_update = ?, input_id = ?, input_hash = ?, input_name = ? WHERE input_directory = ? with args [0, u'sabnzbd', 737035, u'SABnzbd_nzo_mqWyJu', u'SABnzbd_nzo_mqWyJu', u'SHOWNAME.720p.HEVC.x265-MeGusta-xpost.nzb', u'/Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost']
2018-12-07 08:34:55 DB      ::MAIN: nzbtomedia.db: INSERT OR IGNORE INTO downloads (status, client_agent, last_update, input_id, input_hash, input_name, input_directory) VALUES (?, ?, ?, ?, ?, ?, ?) with args [0, u'sabnzbd', 737035, u'SABnzbd_nzo_mqWyJu', u'SABnzbd_nzo_mqWyJu', u'SHOWNAME.720p.HEVC.x265-MeGusta-xpost.nzb', u'/Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost']
2018-12-07 08:34:55 INFO    ::MAIN: Auto-detected SECTION:SickBeard
2018-12-07 08:34:55 DEBUG   ::MAIN: Checking for archives to extract in directory: /Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost
2018-12-07 08:34:55 INFO    ::MAIN: Calling SickBeard:tv to post-process:SHOWNAME.720p.HEVC.x265-MeGusta-xpost.nzb
2018-12-07 08:34:55 DEBUG   ::SERVER: Attempting to connect to server at https://192.168.1.2:6885
2018-12-07 08:34:55 DEBUG   ::SERVER: Server responded at https://192.168.1.2:6885
2018-12-07 08:34:55 INFO    ::MAIN: Attempting to auto-detect tv fork
2018-12-07 08:34:55 INFO    ::MAIN: SickBeard:tv fork auto-detection found custom params {'force_replace': None, 'failed': None, 'process_method': None, 'path': None, 'type': None, 'return_data': None}
2018-12-07 08:34:55 INFO    ::MAIN: SickBeard:tv fork set to custom
2018-12-07 08:34:55 DEBUG   ::EXCEPTION: Replacing file name 6a9ac919046243ce9a5aa960881fea5d with directory name SHOWNAME.720p.HEVC.x265-MeGusta-xpost
2018-12-07 08:34:55 DEBUG   ::EXCEPTION: Replacing file name /Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost/6a9ac919046243ce9a5aa960881fea5d.mkv with download name /Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost/SHOWNAME.720p.HEVC.x265-MeGusta-xpost.mkv
2018-12-07 08:34:55 INFO    ::MAIN: FLATTEN: Flattening directory: /Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost
2018-12-07 08:34:55 DEBUG   ::MAIN: Checking for empty folders in:/Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost
2018-12-07 08:34:55 POSTPROCESS::SICKBEARD: SUCCESS: The download succeeded, sending a post-process request
2018-12-07 08:34:55 DEBUG   ::SICKBEARD: Opening URL: https://192.168.1.2:6885/api/4552ee6cf771c5fa12103cf33f225622/?cmd=postprocess with params: {'nzbName': 'SHOWNAME.720p.HEVC.x265-MeGusta-xpost.nzb', 'failed': '0', 'path': '/Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost', 'type': 'manual', 'return_data': 0}
2018-12-07 08:35:11 DB      ::DB: Updating status of our download SHOWNAME.720p.HEVC.x265-MeGusta-xpost.nzb in the DB to 1
2018-12-07 08:35:11 DB      ::MAIN: nzbtomedia.db: UPDATE downloads SET status=?, last_update=? WHERE input_name=? with args [1, 737035, u'SHOWNAME.720p.HEVC.x265-MeGusta-xpost.nzb']
2018-12-07 08:35:11 INFO    ::CLEANDIRS: Directory /Media/Videos/tv/SHOWNAME.720p.HEVC.x265-MeGusta-xpost still contains 1 unprocessed file(s), skipping ...
2018-12-07 08:35:11 INFO    ::MAIN: The /nzbToMedia/nzbToSickBeard.py script completed successfully.

It looks like medusa is just expecting the procdir?

clinton-hall commented 5 years ago

The error in Medusa indicates that it thinks this is a torrent download not NZB...

The only issue I see is that we are passing in a parameter called nzbName which Medusa is not expecting. The path is passed in correctly. So possibly editing out the following lines might resolve this: https://github.com/clinton-hall/nzbToMedia/blob/master/core/autoProcess/autoProcessTV.py#L189-L190

a10kiloham commented 5 years ago

thanks i'll try

a10kiloham commented 5 years ago

Still fails. I'm not sure it's actually detecting Torrent - i think that's just a bad log message in Medusa. this is with auto set as type and using api key

2018-12-08 13:32:20 DEBUG   ::MAIN: No git specified, can't use git commands
2018-12-08 13:32:20 INFO    ::MAIN: Checking if git needs an update
2018-12-08 13:32:20 DEBUG   ::MAIN: No git specified, can't use git commands
2018-12-08 13:32:20 ERROR   ::MAIN: Unable to determine installed version via git, please check your logs!
2018-12-08 13:32:20 INFO    ::MAIN: No update needed
2018-12-08 13:32:20 INFO    ::MAIN: nzbToMedia Version:11.06 Branch:master (Linux 4.15.0-42-generic)
2018-12-08 13:32:20 WARNING ::MAIN: Failed to locate ffmpeg. Transcoding disabled!
2018-12-08 13:32:20 WARNING ::MAIN: Install ffmpeg with x264 support to enable this feature  ...
2018-12-08 13:32:20 WARNING ::MAIN: Failed to locate ffprobe. Video corruption detection disabled!
2018-12-08 13:32:20 WARNING ::MAIN: Install ffmpeg with x264 support to enable this feature  ...
2018-12-08 13:32:20 DEBUG   ::MAIN: Connecting to transmission: http://192.168.1.2:6883
2018-12-08 13:32:20 INFO    ::MAIN: #########################################################
2018-12-08 13:32:20 INFO    ::MAIN: ## ..::[nzbToMedia.pyc]::.. ##
2018-12-08 13:32:20 INFO    ::MAIN: #########################################################
2018-12-08 13:32:20 DEBUG   ::MAIN: Options passed into nzbToMedia: ['/nzbToMedia/nzbToSickBeard.py', '/Media/Videos/tv/SHOWNAME.1080p.WEB.x264-TBS-xpost', 'SHOWNAME.1080p.WEB.x264-TBS-xpost.nzb', 'SHOWNAME.1080p.WEB.x264-TBS-xpost', '', 'tv', 'alt.binaries.newznzb.delta', '0', '']
2018-12-08 13:32:20 INFO    ::MAIN: Script triggered from SABnzbd 0.7.17+
2018-12-08 13:32:20 DEBUG   ::MAIN: Searching for nzoid from SAbnzbd ...
2018-12-08 13:32:20 DEBUG   ::MAIN: Found nzoid: SABnzbd_nzo_DgGiVf
2018-12-08 13:32:20 DEBUG   ::MAIN: Adding NZB download info for directory /Media/Videos/tv/SHOWNAME.1080p.WEB.x264-TBS-xpost to database
2018-12-08 13:32:20 DB      ::MAIN: nzbtomedia.db: UPDATE downloads SET status = ?, client_agent = ?, last_update = ?, input_id = ?, input_hash = ?, input_name = ? WHERE input_directory = ? with args [0, u'sabnzbd', 737036, u'SABnzbd_nzo_DgGiVf', u'SABnzbd_nzo_DgGiVf', u'SHOWNAME.1080p.WEB.x264-TBS-xpost.nzb', u'/Media/Videos/tv/SHOWNAME.1080p.WEB.x264-TBS-xpost']
2018-12-08 13:32:20 DB      ::MAIN: nzbtomedia.db: INSERT OR IGNORE INTO downloads (status, client_agent, last_update, input_id, input_hash, input_name, input_directory) VALUES (?, ?, ?, ?, ?, ?, ?) with args [0, u'sabnzbd', 737036, u'SABnzbd_nzo_DgGiVf', u'SABnzbd_nzo_DgGiVf', u'SHOWNAME.1080p.WEB.x264-TBS-xpost.nzb', u'/Media/Videos/tv/SHOWNAME.1080p.WEB.x264-TBS-xpost']
2018-12-08 13:32:20 INFO    ::MAIN: Auto-detected SECTION:SickBeard
2018-12-08 13:32:20 DEBUG   ::MAIN: Checking for archives to extract in directory: /Media/Videos/tv/SHOWNAME.1080p.WEB.x264-TBS-xpost
2018-12-08 13:32:20 INFO    ::MAIN: Calling SickBeard:tv to post-process:SHOWNAME.1080p.WEB.x264-TBS-xpost.nzb
2018-12-08 13:32:20 DEBUG   ::SERVER: Attempting to connect to server at https://192.168.1.2:6885
2018-12-08 13:32:20 DEBUG   ::SERVER: Server responded at https://192.168.1.2:6885
2018-12-08 13:32:20 INFO    ::MAIN: Attempting to auto-detect tv fork
2018-12-08 13:32:20 INFO    ::MAIN: SickBeard:tv fork auto-detection found custom params {'force_replace': None, 'failed': None, 'process_method': None, 'path': None, 'type': None, 'return_data': None}
2018-12-08 13:32:20 INFO    ::MAIN: SickBeard:tv fork set to custom
2018-12-08 13:32:20 INFO    ::MAIN: FLATTEN: Flattening directory: /Media/Videos/tv/SHOWNAME.1080p.WEB.x264-TBS-xpost
2018-12-08 13:32:20 DEBUG   ::MAIN: Checking for empty folders in:/Media/Videos/tv/SHOWNAME.1080p.WEB.x264-TBS-xpost
2018-12-08 13:32:20 POSTPROCESS::SICKBEARD: SUCCESS: The download succeeded, sending a post-process request
2018-12-08 13:32:20 DEBUG   ::SICKBEARD: Opening URL: https://192.168.1.2:6885/api/4552ee6cf771c5fa12103cf33f225622/?cmd=postprocess with params: {'failed': '0', 'path': '/Media/Videos/tv/SHOWNAME.1080p.WEB.x264-TBS-xpost', 'type': 'manual', 'return_data': 0}
2018-12-08 13:32:48 DB      ::DB: Updating status of our download SHOWNAME.1080p.WEB.x264-TBS-xpost.nzb in the DB to 1
2018-12-08 13:32:48 DB      ::MAIN: nzbtomedia.db: UPDATE downloads SET status=?, last_update=? WHERE input_name=? with args [1, 737036, u'SHOWNAME.1080p.WEB.x264-TBS-xpost.nzb']
2018-12-08 13:32:48 INFO    ::CLEANDIRS: Directory /Media/Videos/tv/SHOWNAME.1080p.WEB.x264-TBS-xpost still contains 1 unprocessed file(s), skipping ...
2018-12-08 13:32:48 INFO    ::MAIN: The /nzbToMedia/nzbToSickBeard.py script completed successfully.
a10kiloham commented 5 years ago

The main.db entries show the history is expecting just the folder name (proc_dir?) but I can't figure out the right inputs here in the script to make this work.

a10kiloham commented 5 years ago

Setting manually to Medusa still errors. It seems it's passing proc_dir but that includes the leading path - Medusa expects just that folder name?

2018-12-09 13:07:42 DEBUG   ::SICKBEARD: Opening URL: https://192.168.1.2:6885/home/postprocess/processEpisode with params: {'quiet': 1, 'failed': '0', 'proc_dir': '/Media/Videos/tv/SHOW.720p.HEVC.x265-MeGusta-xpost'}
clinton-hall commented 5 years ago

Usually this would take the full path, not just the folder name.

Are Medusa and SABnzbd on different PCs, or in containers/dockers that have different Paths? If so, you need to use the "Mount" options to allow the Path seen by SABnzbd to be translated into the Path seen by Medusa.

a10kiloham commented 5 years ago

It's dockerized but paths are equal in each docker image. but the Medusa log doesn't have any folder data in the database so it seems to be expecting a match of just the folder itself without any extra path

clinton-hall commented 5 years ago

the Medusa log doesn't have any folder data in the database so it seems to be expecting a match of just the folder itself without any extra path

Can you please clarify this? Can you show the Medusa log?

a10kiloham commented 5 years ago

I mean the lines in the history in the main.db sqlite file. Upon snatch it adds the name but this returns full directory path.

clinton-hall commented 5 years ago

Ok, the snatch database entries shouldn't have any relevance to the download complete scan.

The way the script works is it tells Medusa that the download has complete and where the files are to process.

When processing via api this passes path and not proc_dir, when using the web UI it uses proc_dir

what happens if you put the following into your browser? https://192.168.1.2:6885/home/postprocess/processEpisode?quiet=1&failed=0&proc_dir=/Media/Videos/tv/SHOW.720p.HEVC.x265-MeGusta-xpost

a10kiloham commented 5 years ago

In this case the processEpisode handler returns

 Processing path: /Media/Videos/tv/SHOWNAME.720p.AMZN.WEBRip.DDP5.1.x264-NTb-xpost
Processing folder: /Media/Videos/tv/SHOWNAME.720p.AMZN.WEBRip.DDP5.1.x264-NTb-xpost
Post-processing files: [u'SHOWNAME.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.mkv']
Post-processing video files: [u'SHOWNAME.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.mkv']
Processing /Media/Videos/tv/SHOWNAME.720p.AMZN.WEBRip.DDP5.1.x264-NTb-xpost/SHOWNAME.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.mkv
Show information parsed from file name
Retrieving episode object for Sweet Home S02E06
The episode file has a quality in it, using that: 720p WEB-DL
This episode was found in history as SNATCHED.
Snatch in history: True
Manually snatched: False
Info hash: None
NZB: False
Current quality: 720p WEB-DL
New quality: 720p WEB-DL
Proper: False
This episode is a priority download: True
New file: /Media/Videos/tv/SHOWNAME.720p.AMZN.WEBRip.DDP5.1.x264-NTb-xpost/SHOWNAME.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.mkv
New size: 1.27 GB
There is no existing file so there's no worries about replacing it
This download is marked a priority download so I'm going to replace an existing file if I find one
Found release name SHOWNAME.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb
Destination folder for this episode: /mnt/Resources/TV/Sweet Home Oklahoma/Season.02
No associated files were found for /Media/Videos/tv/SHOWNAME.720p.AMZN.WEBRip.DDP5.1.x264-NTb-xpost/SHOWNAME.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.mkv
Moving then symbolic linking file from /Media/Videos/tv/SHOWNAME.720p.AMZN.WEBRip.DDP5.1.x264-NTb-xpost/SHOWNAME.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.mkv to /mnt/Resources/TV/Sweet Home Oklahoma/Season.02/Sweet.Home.S02E06.Judge, Jury, and Inspector.720p WEB-DL.mkv
Processing succeeded for /Media/Videos/tv/SHOWNAME.720p.AMZN.WEBRip.DDP5.1.x264-NTb-xpost/SHOWNAME.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.mkv
Successfully processed.

and Medusa sends warning:

2018-12-10 09:51:27 WARNING  Thread_18 :: [54da0c7] Please consider manually move torrent to seed folder as there is no info hash in snatch history: /Media/Videos/tv/SHOWNAME.720p.AMZN.WEBRip.DDP5.1.x264-NTb-xpost/SHOWNAME.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.mkv
a10kiloham commented 5 years ago

Should there be a name parameter sent to Medusa?

medariox commented 5 years ago

Yes, try with:

https://192.168.1.2:6885/home/postprocess/processEpisode?quiet=1&failed=0&proc_dir=/Media/Videos/tv/SHOW.720p.HEVC.x265-MeGusta-xpost&nzbName=SHOW.720p.HEVC.x265-MeGusta-xpost
clinton-hall commented 5 years ago

Ok, as it shows here, the issue is that Medusa is saying this is a "torrent" and is therefore doing the symlinking etc.

So why is this SABnzbd download being detected as a Torrent? Was this snatched by Medusa?

I Do also note that the api did have the nzbName parameter passed, and this still have the Please consider manually move torrent to seed folder as there is no info hash in snatch history warning...

a10kiloham commented 5 years ago

Manually setting the fork to medusa at the moment and removing the API key seems to be working. I'll test it out a few more days.