search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.86k stars 976 forks source link

[deviantart] HTTP request failed on refreshing private token #442

Closed ST-Lawson closed 4 years ago

ST-Lawson commented 5 years ago

I've set up an "app" in DeviantArt and gotten the valid client_id and client_secret, and did the oauth authentication and received a refresh token, however I am receiving the following error when I try to get anything from DeviantArt:

[deviantart][info] Refreshing private access token [deviantart][error] HTTP request failed: 400: Bad Request for url: https://www.deviantart.com/oauth2/token

In my config file, the relevant section is as follows:

"deviantart": { "refresh-token": "123456789123456789123456789", "client-id": "XXXXX", "client-secret": "abcdefghijklmnopqrstuvwxyz" },

Any idea what might be happening? Thanks in advance for any help.

mikf commented 5 years ago

Did you do the OAuth authentication step after setting up your own client-id and client-secret in your config file? Because your refresh-token will be tied to those two values, and getting a token before using your own API credentials will get you one that is only valid for the default "App".

ST-Lawson commented 5 years ago

Got a new token (while logged into account that has the client-id and client-secret set up). Error is now: [deviantart][info] Refreshing private access token [deviantart][error] HTTP request failed: 401: Unauthorized for url: https://www.deviantart.com/oauth2/token

Does the order of elements under "deviantart" matter? Does the refresh token need to be after the client-id and client-secret in the order?

mikf commented 5 years ago

Does the order of elements under "deviantart" matter?

No, the order doesn't matter.

I can reproduce the same error ("401: Unauthorized") by setting a wrong client-secret value, but then I also can't get a new refresh-token with gallery-dl oauth:deviantart ("invalid_client") and it also fails on the "Requesting public access token" step, so that can't be it. Getting a public token works for you, right?

I also just tried renewing my refresh-token and everything works. Here is what I did:

ST-Lawson commented 5 years ago

Ok, followed your directions. While logged into my DA account, checked my api credentials to make sure they were good. Removed the refresh token from my config file to try it just with the credentials. Ran it and that seemed to work ok. I run the oauth:deviantart on it, takes me directly to a localhost:6414/?code=somestuff page that says "Your Refresh Token is..." and has my token along with the directions for how to put it in the config file.

Run it with the refresh token added and get the following: [gallery-dl][debug] Version 1.10.5 [gallery-dl][debug] Python 3.7.3 - Windows-10-10.0.18362 [gallery-dl][debug] requests 2.22.0 - urllib3 1.25.6 [gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/kitty-chancatorie' [gallery-dl][debug] Updating urllib3 ciphers [deviantart][debug] Using custom API credentials (client-id 10581) [deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/kitty-chancatorie' [urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443 [urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/kitty-chancatorie HTTP/1.1" 200 2677 [urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=kitty-chancatorie&offset=0&limit=24&mature_content=true HTTP/1.1" 200 5709 [deviantart][debug] Switching to private access token [deviantart][info] Refreshing private access token [urllib3.connectionpool][debug] https://www.deviantart.com:443 "POST /oauth2/token HTTP/1.1" 400 97 [deviantart][error] HTTP request failed: 400: Bad Request for url: https://www.deviantart.com/oauth2/token

For testing purposes, I ran it on the user account that you appeared to use (shimoda7) and it did seem to work through the photos, although kept giving me errors on them (which appears to be a known existing bug: https://github.com/mikf/gallery-dl/issues/436 So maybe it's something with that specific account that's wonky?

I do really appreciate this utility. I've used it without issue on Instagram and Flickr, and I understand how tricky it can be working with DA's api (I wrote something similar but in php on my webserver and I'm hoping to use this as a more "local" option).

mikf commented 5 years ago

I've tested with https://www.deviantart.com/kitty-chancatorie and it works just fine on my machine, so it's not that specific account. I've also tried a bunch of other stuff (new dA account, new application) to see if maybe DeviantArt is blocking newly registered applications or something else, but that worked as well.

The last thing that comes to mind would be a disabled or non-persistent cache file on your end. DeviantArt hands out a new refresh token every time one is used, so gallery-dl stores those in its cache file and uses the initial token as lookup key. Deleting this file invalidates the "refresh token chain" and you'd have to get a new initial token, although the first "Refreshing private access token" step should work regardless. If even that doesn't work, than that's not the problem either.

There also were, at some point, more descriptive error messages for OAuth2 authentication failures, but I've somehow managed to disable them. https://github.com/mikf/gallery-dl/commit/df2b3c68881194d1a188325e3be3371f1c16fec2 restores them again. Could you get these changes and post what kind of error message you get? With -v enabled, if possible.

tdh-ba commented 5 years ago

I've had the exact same issue twice, so I tried to download from several accounts to try and pinpoint the problem. It didn't work out, all I can say now is "I have that too and it's weird". Sometimes it worked, sometimes I got the HTTP request failed: 400, seemingly at random. Retrying, with or without clearing the cache, makes no difference.

And what always seems to fail is refreshing the access token while downloading an entire gallery.

I wonder if it's a problem on the other end. Even DA's own mobile app has been wonkier than usual for at least the past week, eating notes and submissions, displaying only half of a gallery's content, showing folders as empty that very much aren't etc.

Ailothaen commented 5 years ago

Hello, I am running into the same issue as well.

Here is my output log:

$ gallery-dl.exe -g -c .\gallery-dl.conf https://www.deviantart.com/ailothaen/gallery/ -v
[gallery-dl][debug] Version 1.10.6
[gallery-dl][debug] Python 3.7.3 - Windows-10-10.0.17763
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.6
[gallery-dl][debug] Starting UrlJob for 'https://www.deviantart.com/ailothaen/gallery/'
[gallery-dl][debug] Updating urllib3 ciphers
[deviantart][debug] Using custom API credentials (client-id 10734)
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/ailothaen/gallery/'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/ailothaen HTTP/1.1" 200 1662
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=ailothaen&offset=0&limit=24&mature_content=true HTTP/1.1" 200 1481
[deviantart][debug] Switching to private access token
[deviantart][info] Refreshing private access token
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "POST /oauth2/token HTTP/1.1" 401 97
[deviantart][error] HTTP request failed:  401: Unauthorized for url: https://www.deviantart.com/oauth2/token

My config file:

{
    "extractor":
    {
        "deviantart": {
            "client-id": 10734,
            "client-secret": "(undisclosed)",
            "refresh-token": "(undisclosed)"
        }
    },
    "cache": {
        "file": "D:/PATH/gallery-dl/cache.sqlite3"
    }
}

The cache file still gets created, and I can get renew-tokens normally. I tried to renew my refresh-token several times and update the line in the config file, but it does not change anything.

mikf commented 5 years ago

I've tried in several ways and, again, can't reproduce this error. Not with the .exe version, not on a Windows machine, not with Wine.

As mentioned before, version 1.10.6 only has limited debug output for those OAuth2 errors, so I've built a pre-release .exe version that includes https://github.com/mikf/gallery-dl/commit/df2b3c68881194d1a188325e3be3371f1c16fec2 and should get some more descriptive error messages: gallery-dl-dev.exe. Could you run this again (with -v) and post the log?

Also the error here seems to be different than the original one (401 Unauthorized <> 400 Bad Request)

tdh-ba commented 5 years ago

I don't want to jinx it, but the issue (always 400, never 401) has disappeared for me now. I didn't change anything, I just didn't touch the downloader for a few days, then tried again.

Ailothaen commented 5 years ago

Here is the answer with gallery-dl-dev.exe :

$ gallery-dl-dev.exe -g -c .\gallery-dl.conf https://www.deviantart.com/ailothaen/gallery/ -v
[gallery-dl][debug] Version 1.11.0-dev
[gallery-dl][debug] Python 3.7.3 - Windows-10-10.0.17763
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.6
[gallery-dl][debug] Starting UrlJob for 'https://www.deviantart.com/ailothaen/gallery/'
[gallery-dl][debug] Updating urllib3 ciphers
[deviantart][debug] Using custom API credentials (client-id 10734)
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/ailothaen/gallery/'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/ailothaen HTTP/1.1" 200 1662
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=Ailothaen&offset=0&limit=24&mature_content=true HTTP/1.1" 200 1481
[deviantart][debug] Switching to private access token
[deviantart][info] Refreshing private access token
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "POST /oauth2/token HTTP/1.1" 401 97
[deviantart][debug] Server response: {'error': 'invalid_request', 'error_description': 'The refresh_token is invalid.', 'status': 'error'}
[deviantart][error] Authentication failed: "The refresh_token is invalid." (invalid_request)

So it looks like there is a problem with the refresh_token. However, I did the oauth:deviantart part before, and replaced the refresh-token in my configuration file (as above), and did not run anything then before the log above.

It should be maybe noted that cache.sqlite3 (see my config file) still gets created, and here is the content: Content of cache database (I recognized the refresh tokens I got, because I did three times that process of requesting a token and then trying the log above, to make sure it was not bad luck)

Also, I don't know if this has an importance, but I did not publish the developer app I created, despite I still filled everything else: Content of cache database

mikf commented 5 years ago

It's weird that you get a 401 status code but an otherwise identical error message then when I try it with an invalid refresh-token. (It's 400 for me)

$ gallery-dl https://www.deviantart.com/ailothaen/gallery/ -o refresh-token=invalid -v
...
[deviantart][info] Refreshing private access token
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "POST /oauth2/token HTTP/1.1" 400 97
[deviantart][debug] Server response: {'error': 'invalid_request', 'error_description': 'The refresh_token is invalid.', 'status': 'error'}
[deviantart][error] AuthenticationError: "The refresh_token is invalid." (invalid_request)

I'm grasping at straws here, but could you try this whole ordeal again with default API credentials and disabled cache? Something like

$ gallery-dl --ignore-config -o cache.file= oauth:deviantart
$ gallery-dl --ignore-config -o cache.file= -o refresh-token="token from above" https://www.deviantart.com/ailothaen/gallery/

The token will only work once, but that's fine for testing purposes.

It should be maybe noted that cache.sqlite3 (see my config file) still gets created

That's expected. The file gets created whenever the cache.py module is imported, not just when a value gets actually cached.

Also, I don't know if this has an importance, but I did not publish the developer app I created

That's fine. My test applications are not published as well and it works regardless. I wouldn't be surprised if even the "main" application is unpublished.

Ailothaen commented 5 years ago

The two commands above are working. So I guess the problem should be around my client-id and client-secret?

$ gallery-dl --ignore-config -o cache.file= oauth:deviantart
Waiting for response. (Cancel with Ctrl+c)

Your Refresh Token is

9ffa9bb53***

Put this value into your configuration file as
'extractor.deviantart.refresh-token'.

Example:
{
    "extractor": {
        "deviantart": {
            "refresh-token": "9ffa9bb53***"
        }
    }
}
gallery-dl --ignore-config -o cache.file= -o refresh-token="9ffa9bb53***" https://www.deviantart.com/ailothaen/gallery/
[deviantart][info] Requesting public access token
[deviantart][info] Refreshing private access token
* .\gallery-dl\deviantart\ailothaen\deviantart_778178998_Ludara painting.jpg
* .\gallery-dl\deviantart\ailothaen\deviantart_737671860_The Lost Ribbon.pdf
mikf commented 5 years ago

Since you seem to explicitly add your API credentials with -c .\gallery-dl.conf every time you want to download from DeviantArt , could it be that you're not doing that when getting a new refresh-token?

github-userx commented 5 years ago

@mikf can you push the newest version of gallery-dl to pip? I’m only getting 1.10.6 which still gives me Imgur errors (nsfw)

mikf commented 5 years ago

@github-userx You can install the master branch version with

pip install --upgrade https://github.com/mikf/gallery-dl/archive/master.zip
github-userx commented 5 years ago

Thanks 

Ailothaen commented 5 years ago

Oh yeah you are right. I was making the oauth process without specifying the configuration file option, and so my credentials were missing. Now that I included them, the window I get makes so more sense (I got a prompt about authorizing the app to my account) Thanks for the help!