DownloadStation API version has changed since DSM 7.0

MrCee commented 3 years ago

Technical Specs

Running on 'Synology DSM 7.0'
Python version '3.8+'
Download Client 'Download Station'
Intended Media Management 'Sickchill/Radarr'

Error encounted: [18:39:49] [ERROR]::MAIN: Failed to connect to synology

Hi Clinton,

I've just upgraded to DSM 7.0 and notice the API version needs to be increased, there were 2 lines hardcoded in libs/custom/syno/auth.py where 'version': '2' could now potentially be updated to anything between 3-7 as I have confirmed working on DSM 7.0.

I'm not sure what effect this will have on DSM 6.2 users but thought maybe you want to get the max version from SYNO.API.Auth which is the first row returned from the following:

        def get_api_list(self, app=None):
        query_path = 'query.cgi?api=SYNO.API.Info'
        list_query = {'version': '1', 'method': 'query', 'query': 'all'}

this goes for 650+ rows...

{"data":{"SYNO.API.Auth":{"maxVersion":7,"minVersion":1,"path":"entry.cgi"},"SYNO.API.Auth.Key":{"maxVersion":7,"minVersion":7,"path":"entry.cgi","requestFormat":"JSON"},"SYNO.API.Auth.Key.Code":{"maxVersion":7,"minVersion":7,"path":"entry.cgi","requestFormat":"JSON"},"SYNO.API.Auth.RedirectURI":{"maxVersion":1,"minVersion":1,"path":"entry.cgi","requestFormat":"JSON"},"SYNO.API.Auth.Type":

The below query parameters were taken from the old API guide, you could minimise the list query to 2 rows (I'm not sure if anything else is needed from the 650+ rows, still learning the code):

        def get_api_list(self, app=None):
        query_path = 'query.cgi?api=SYNO.API.Info'
        list_query = {'version': '1', 'method': 'query', 'query': 'SYNO.API.Auth,SYNO.DownloadStation.Task'}

JSON output is:

{"data":{"SYNO.API.Auth":{"maxVersion":7,"minVersion":1,"path":"entry.cgi"},"SYNO.DownloadStation.Task":{"maxVersion":3,"minVersion":1,"path":"DownloadStation/task.cgi"}},"success":true}

Let me know if there's anything I can test.

Cheers

MrCee commented 3 years ago

I have just installed DSM 6.2 VM now so I am able to test this.

If we were to go with the method of getting the maxVersion of SYNO.API.Info from the results of the list_query (rather than hardcoding API version), this is confirmed as working....

        def get_api_list(self, app=None):
        query_path = 'query.cgi?api=SYNO.API.Info'
        list_query = {'version': '1', 'method': 'query', 'query': 'SYNO.API.Auth,SYNO.DownloadStation.Task'}

JSON output for DSM 6.2 is:

{"data":{"SYNO.API.Auth":{"maxVersion":6,"minVersion":1,"path":"auth.cgi"},"SYNO.DownloadStation.Task":{"maxVersion":3,"minVersion":1,"path":"DownloadStation/task.cgi"}},"success":true}

JSON output for DSM 7.0 is:

{"data":{"SYNO.API.Auth":{"maxVersion":7,"minVersion":1,"path":"entry.cgi"},"SYNO.DownloadStation.Task":{"maxVersion":3,"minVersion":1,"path":"DownloadStation/task.cgi"}},"success":true}

clinton-hall commented 3 years ago

I think I have added something that should do it...

Can you switch to Syno-api branch and test?

In theory this should get the maxVersion for the SYNO.API.Auth and then login with that.

I am surprised that this causes issues, as the minVersion is 1, so hard-coding Version = 2 shouldn't have stopped the login... All other calls do actually call the maxVersion for the specified command.

MrCee commented 3 years ago

Many undocumented changes with DSM 7.0 and Download Station (Transmission under the hood) are for the better especially when it comes to running external scripts and working around AppArmor (wiki updated) ... getting this API version correct would be the final hurdle. I'm surprised also... API V2 or below simply do not work with DSM 7.0

At present Syno-api branch did not work. if I hardcoded libs/custom/syno/auth.py to 'version': '6' for DSM 6.2 or 'version': '7' for DSM 7.0 it would work as usual.

clinton-hall commented 3 years ago

ok... that makes sense (regardless of the "minVersion" later versions do not support V2 or below...).

I think the issue was that the check for max version was erroring.... I submitted a quick fix. Can you update and test again (Syno-api branch).

Just really annoying that this is all a theoretical change for me as I can't test DSM here.

clinton-hall commented 3 years ago

Also... If you are able to find any more details in the logs/errors etc that might indicate where it fails. Tracebacks etc?

MrCee commented 3 years ago

Happy to help with a new Synology NAS and VM's to boot :) You did it, it works! Confirmed testing on DSM 6.2 & DSM 7.0 Thank you.

clinton-hall commented 3 years ago

awesome.... so, just 1 little typo. For a theoretical change, I'll take that as a win!

MrCee commented 3 years ago

yes! awesome! this is great!

clinton-hall commented 3 years ago

Great, I am syncing up to nightly (Syno-api branch will disappear shortly).

Let me know if you encounter any issues... all the stop, pause, resume, delete should work (these already check the maxVersion)...

MrCee commented 3 years ago

I overlooked this in earlier testing... Invalid number of arguments received from client

Although we have successful connections again and the script runs... the torrent wasn't getting removed from Download Station upon completing, and I just noticed this would be due to the script switching to a manual run. When you have time, are you able to take a look and let me know if I can test or show you an API response if that helps? Thank you

2021-08-25 23:12:14 INFO    ::MAIN: Loading config from [/volume1/docker/nzbToMedia/autoProcessMedia.cfg]
2021-08-25 23:12:14 INFO    ::ENVIRONMENT: PATH: /usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/syno/sbin:/usr/syno/bin:/usr/local/sbin
2021-08-25 23:12:14 INFO    ::ENVIRONMENT: TR_APP_VERSION: 2.93
2021-08-25 23:12:14 INFO    ::ENVIRONMENT: TR_TIME_LOCALTIME: Wed Aug 25 23:12:10 2021
2021-08-25 23:12:14 INFO    ::ENVIRONMENT: TR_TORRENT_DIR: /var/services/download/664
2021-08-25 23:12:14 INFO    ::ENVIRONMENT: TR_TORRENT_HASH: 57a7a4803d20b441f58326b2e6b65cfba0f14607
2021-08-25 23:12:14 INFO    ::ENVIRONMENT: TR_TORRENT_ID: 664
2021-08-25 23:12:14 INFO    ::ENVIRONMENT: TR_TORRENT_NAME: Wentworth S09E01 - 1080p WEB-DL Nemo[eztv.re].mkv
2021-08-25 23:12:14 INFO    ::MAIN: Python v3.8 will reach end of life in 1146 days.
2021-08-25 23:12:14 INFO    ::MAIN: Checking database structure...
2021-08-25 23:12:14 DEBUG   ::MAIN: Checking Initial Schema database upgrade
2021-08-25 23:12:14 DB      ::MAIN: nzbtomedia.db: SELECT 1 FROM sqlite_master WHERE name = ?; with args ('db_version',)
2021-08-25 23:12:14 DB      ::MAIN: nzbtomedia.db: SELECT db_version FROM db_version
2021-08-25 23:12:14 DEBUG   ::MAIN: InitialSchema upgrade not required
2021-08-25 23:12:14 DEBUG   ::MAIN: Checking if we can use git commands: "/usr/local/bin/git" version
2021-08-25 23:12:14 DEBUG   ::MAIN: Executing "/usr/local/bin/git" version with your shell in /volume1/docker/nzbToMedia
2021-08-25 23:12:14 DEBUG   ::MAIN: git output: git version 2.32.0
2021-08-25 23:12:14 DEBUG   ::MAIN: "/usr/local/bin/git" version : returned successful
2021-08-25 23:12:14 DEBUG   ::MAIN: Using: "/usr/local/bin/git"
2021-08-25 23:12:14 DEBUG   ::MAIN: Executing "/usr/local/bin/git" symbolic-ref -q HEAD with your shell in /volume1/docker/nzbToMedia
2021-08-25 23:12:14 DEBUG   ::MAIN: git output: refs/heads/nightly
2021-08-25 23:12:14 DEBUG   ::MAIN: "/usr/local/bin/git" symbolic-ref -q HEAD : returned successful
2021-08-25 23:12:14 INFO    ::MAIN: Checking if git needs an update
2021-08-25 23:12:14 DEBUG   ::MAIN: Executing "/usr/local/bin/git" rev-parse HEAD with your shell in /volume1/docker/nzbToMedia
2021-08-25 23:12:14 DEBUG   ::MAIN: git output: 411e70ba92ff25df6bbf960974dbd46993e136ce
2021-08-25 23:12:14 DEBUG   ::MAIN: "/usr/local/bin/git" rev-parse HEAD : returned successful
2021-08-25 23:12:14 DEBUG   ::MAIN: Executing "/usr/local/bin/git" fetch origin with your shell in /volume1/docker/nzbToMedia
2021-08-25 23:12:15 DEBUG   ::MAIN: git output: 
2021-08-25 23:12:15 DEBUG   ::MAIN: "/usr/local/bin/git" fetch origin : returned successful
2021-08-25 23:12:15 DEBUG   ::MAIN: Executing "/usr/local/bin/git" rev-parse --verify --quiet '@{upstream}' with your shell in /volume1/docker/nzbToMedia
2021-08-25 23:12:15 DEBUG   ::MAIN: git output: 411e70ba92ff25df6bbf960974dbd46993e136ce
2021-08-25 23:12:15 DEBUG   ::MAIN: "/usr/local/bin/git" rev-parse --verify --quiet '@{upstream}' : returned successful
2021-08-25 23:12:15 DEBUG   ::MAIN: Executing "/usr/local/bin/git" rev-list --left-right '@{upstream}'...HEAD with your shell in /volume1/docker/nzbToMedia
2021-08-25 23:12:15 DEBUG   ::MAIN: git output: 
2021-08-25 23:12:15 DEBUG   ::MAIN: "/usr/local/bin/git" rev-list --left-right '@{upstream}'...HEAD : returned successful
2021-08-25 23:12:15 DEBUG   ::MAIN: cur_commit = 411e70ba92ff25df6bbf960974dbd46993e136ce % (newest_commit)= 411e70ba92ff25df6bbf960974dbd46993e136ce, num_commits_behind = 0, num_commits_ahead = 0
2021-08-25 23:12:15 INFO    ::MAIN: No update needed
2021-08-25 23:12:15 INFO    ::MAIN: nzbToMedia Version:411e70ba92ff25df6bbf960974dbd46993e136ce Branch:nightly (Linux 4.4.180+)
2021-08-25 23:12:15 DEBUG   ::MAIN: Connecting to synology: http://localhost:5000
2021-08-25 23:12:17 INFO    ::MAIN: #########################################################
2021-08-25 23:12:17 INFO    ::MAIN: ## ..::[TorrentToMedia.py]::.. ##
2021-08-25 23:12:17 INFO    ::MAIN: #########################################################
2021-08-25 23:12:17 DEBUG   ::MAIN: Options passed into TorrentToMedia: ['/volume1/docker/nzbToMedia/TorrentToMedia.py']
2021-08-25 23:12:17 WARNING ::MAIN: Invalid number of arguments received from client, Switching to manual run mode ...
2021-08-25 23:12:17 INFO    ::MAIN: Searching /volume1/downloads/movie for mediafiles to post-process ...
2021-08-25 23:12:17 INFO    ::MAIN: Searching /volume1/downloads/movie for mediafiles to post-process ...
2021-08-25 23:12:17 DEBUG   ::MAIN: No directories identified in Radarr:movie for post-processing
2021-08-25 23:12:17 INFO    ::MAIN: Searching /volume1/downloads/tv for mediafiles to post-process ...
2021-08-25 23:12:17 DEBUG   ::MAIN: Found file Wentworth S09E01 - 1080p WEB-DL Nemo[eztv.re].mkv in root directory /volume1/downloads/tv.
2021-08-25 23:12:17 INFO    ::COPYLINK: MEDIAFILE: [b'Wentworth S09E01 - 1080p WEB-DL Nemo[eztv.re].mkv']
2021-08-25 23:12:17 INFO    ::COPYLINK: SOURCE FOLDER: [/volume1/downloads/tv]
2021-08-25 23:12:17 INFO    ::COPYLINK: TARGET FOLDER: [b'/volume1/downloads/tv/Wentworth']
2021-08-25 23:12:17 INFO    ::COPYLINK: Hard linking SOURCE MEDIAFILE -> TARGET FOLDER
2021-08-25 23:12:17 INFO    ::MAIN: Searching /volume1/downloads/tv for mediafiles to post-process ...
2021-08-25 23:12:17 DEBUG   ::MAIN: Found file Wentworth S09E01 - 1080p WEB-DL Nemo[eztv.re].mkv in root directory /volume1/downloads/tv.
2021-08-25 23:12:17 INFO    ::COPYLINK: MEDIAFILE: [b'Wentworth S09E01 - 1080p WEB-DL Nemo[eztv.re].mkv']
2021-08-25 23:12:17 INFO    ::COPYLINK: SOURCE FOLDER: [/volume1/downloads/tv]
2021-08-25 23:12:17 INFO    ::COPYLINK: TARGET FOLDER: [b'/volume1/downloads/tv/Wentworth']
2021-08-25 23:12:17 INFO    ::COPYLINK: MEDIAFILE already exists in the TARGET folder, skipping ...
2021-08-25 23:12:17 INFO    ::MAIN: Starting manual run for SickBeard:tv - Folder:/volume1/downloads/tv/Wentworth
2021-08-25 23:12:17 INFO    ::MAIN: Checking database for download info for Wentworth ...
2021-08-25 23:12:17 DB      ::DB: Getting download info for Wentworth from the DB
2021-08-25 23:12:17 DB      ::MAIN: nzbtomedia.db: SELECT * FROM downloads WHERE input_name=? AND status=? with args ['Wentworth', 0]
2021-08-25 23:12:17 INFO    ::MAIN: Unable to locate download info for Wentworth, continuing to try and process this release ...
2021-08-25 23:12:17 DEBUG   ::MAIN: Received Directory: /volume1/downloads/tv/Wentworth | Name: Wentworth | Category: tv
2021-08-25 23:12:17 DEBUG   ::MAIN: SEARCH: Found the Category: tv in directory structure
2021-08-25 23:12:17 INFO    ::MAIN: SEARCH: Found a unique directory Wentworth in the category directory
2021-08-25 23:12:17 DEBUG   ::MAIN: Determined Directory: /volume1/downloads/tv/Wentworth | Name: Wentworth | Category: tv
2021-08-25 23:12:17 INFO    ::MAIN: Auto-detected SECTION:SickBeard
2021-08-25 23:12:17 INFO    ::MAIN: Output directory set to: /volume1/downloads/tv/Wentworth
2021-08-25 23:12:17 DEBUG   ::MAIN: Scanning files in directory: /volume1/downloads/tv/Wentworth
2021-08-25 23:12:17 DEBUG   ::MAIN: Found 1 files in /volume1/downloads/tv/Wentworth
2021-08-25 23:12:17 INFO    ::COPYLINK: MEDIAFILE: [Wentworth S09E01 - 1080p WEB-DL Nemo[eztv.re].mkv]
2021-08-25 23:12:17 INFO    ::COPYLINK: SOURCE FOLDER: [/volume1/downloads/tv/Wentworth]
2021-08-25 23:12:17 INFO    ::COPYLINK: TARGET FOLDER: [/volume1/downloads/tv/Wentworth]
2021-08-25 23:12:17 INFO    ::COPYLINK: SOURCE AND TARGET files are the same, skipping ...
2021-08-25 23:12:17 DEBUG   ::MAIN: Checking for archives to extract in directory: /volume1/downloads/tv/Wentworth
2021-08-25 23:12:17 INFO    ::MAIN: FLATTEN: Flattening directory: /volume1/downloads/tv/Wentworth
2021-08-25 23:12:17 DEBUG   ::MAIN: Checking for empty folders in:/volume1/downloads/tv/Wentworth
2021-08-25 23:12:17 INFO    ::MAIN: Found 1 media files in /volume1/downloads/tv/Wentworth
2021-08-25 23:12:17 INFO    ::MAIN: Calling SickBeard:tv to post-process:Wentworth
2021-08-25 23:12:17 INFO    ::MAIN: Changing file mode of /volume1/downloads/tv/Wentworth to 0o777
2021-08-25 23:12:17 DEBUG   ::SERVER: Attempting to connect to server at http://172.20.0.1:8081
2021-08-25 23:12:17 DEBUG   ::SERVER: Server responded at http://172.20.0.1:8081
2021-08-25 23:12:17 INFO    ::MAIN: Attempting to auto-detect tv fork
2021-08-25 23:12:17 DEBUG   ::MAIN: Removing excess parameters: ['delete_files', 'delete_on', 'dir', 'dirName', 'ignore_subs', 'resource']
2021-08-25 23:12:17 INFO    ::MAIN: SickBeard:tv fork auto-detection successful ...
2021-08-25 23:12:17 INFO    ::MAIN: SickBeard:tv fork set to SickChill-api
2021-08-25 23:12:17 DEBUG   ::MAIN: Create object for fork SickChill-api
2021-08-25 23:12:17 INFO    ::MAIN: SickBeard:tv Could not create a fork object for SickChill-api. Probaly class not added yet.
2021-08-25 23:12:17 DEBUG   ::MAIN: calling command: /usr/local/ffmpeg/bin/ffprobe -v quiet -print_format json -show_format -show_streams -show_error /volume1/docker/nzbToMedia/tests/test.mp4
2021-08-25 23:12:17 INFO    ::TRANSCODER: Checking [Wentworth S09E01 - 1080p WEB-DL Nemo[eztv.re].mkv] for corruption, please stand by ...
2021-08-25 23:12:17 DEBUG   ::MAIN: calling command: /usr/local/ffmpeg/bin/ffprobe -v quiet -print_format json -show_format -show_streams -show_error /volume1/downloads/tv/Wentworth/Wentworth S09E01 - 1080p WEB-DL Nemo[eztv.re].mkv
2021-08-25 23:12:17 INFO    ::TRANSCODER: SUCCESS: [Wentworth S09E01 - 1080p WEB-DL Nemo[eztv.re].mkv] has no corruption.
2021-08-25 23:12:17 POSTPROCESS::SICKBEARD: SUCCESS: The download succeeded, sending a post-process request
2021-08-25 23:12:17 DEBUG   ::SICKBEARD: Opening URL: http://172.20.0.1:8081/api/2688c9b7b8228ba8dcec2c37f06c1bad/ with params: {'path': '/volume1/downloads/tv/Wentworth', 'proc_dir': '/volume1/downloads/tv/Wentworth', 'failed': 0, 'process_method': 'move', 'force': 1, 'force_replace': 1, 'return_data': 0, 'delete': 1, 'force_next': 1, 'cmd': 'postprocess', 'nzbName': 'Wentworth'}
2021-08-25 23:12:43 DEBUG   ::PLEX: Attempting to update Plex Library for category tv.
2021-08-25 23:12:43 DEBUG   ::PLEX: Plex Library has been refreshed.
2021-08-25 23:12:43 INFO    ::CLEANDIR: Directory /volume1/downloads/tv/Wentworth has been processed and removed ...
2021-08-25 23:12:43 INFO    ::MAIN: The /volume1/docker/nzbToMedia/TorrentToMedia.py script completed successfully.

MrCee commented 3 years ago

I have added the Environment Variables to the above log.

MrCee commented 3 years ago

I found the problem, just learning git commands while I send through a pull request!

clinton-hall commented 3 years ago

Thanks for the PR... A few things to note:

Send the PR against Nightly, not Master
Don't change permissions on all files (you will see the PR shows 1925 files changed when I believe you only changed 3 lines in the syno/auth.py file?

Otherwise, looks like you figured it out... I'll update Nightly soon.

clinton-hall commented 3 years ago

https://github.com/clinton-hall/nzbToMedia/commit/36eddcfb923a68b67501d8856f62f66c3e6cbcbc

Let me know if I'm missing anything

MrCee commented 3 years ago

Definitely with you on PR against Nightly, not Master (Ive seen you mention it countless times)... I totally ballsed it up, and now see where I went wrong. I'll try and give the pull request another go next time if I can. But for now, everything is working as it should, I have 1 instance of the nzbToMedia folder working with multiple docker containers and Synology apps with no issue passing in/out of containers, which one day I'll have to get round to adding to a wiki. Thank you.

MrCee commented 3 years ago

This is working really well and would be good to go in Master.

clinton-hall / nzbToMedia

DownloadStation API version has changed since DSM 7.0 #1840