search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.46k stars 939 forks source link

Deviant Art gives 'API responded with 429 Too Many Requests' constantly, impossible to download #5794

Closed Brad3273 closed 2 months ago

Brad3273 commented 3 months ago

Is it at all possible to download much from Deviant Art these days?

I was downloading pictures with no problems, using gallery-dl 1.26.8, until a couple of months ago. Now I get immediate 'API responded with 429 Too Many Requests' warnings and nothing downloads.

I've looked at other issues here which mention the problem, I believe I've properly set up my API credentials and refresh-token, but the issue persists.

I can use Deviant Art 'manually' via browser with no issues, on the same IP address and account credentials.

I've got listings of configuration files and output below; I'd appreciate any help.

First of all I tried a very simple configuration file using public credentials:

{
    "extractor":    {   
        "deviantart":   {
            "skip": true,
            "public":   true
        }
    }
}

This was the output when I ran gallery-dl against a public gallery:

bash-4.2$ gallery-dl -v --sleep 10 "https://www.deviantart.com/necromancer-rev/gallery/"
[gallery-dl][debug] Version 1.26.8
[gallery-dl][debug] Python 3.6.8 - Linux-4.4.2-x86_64-with-centos-7.9.2009-Core
[gallery-dl][debug] requests 2.27.1 - urllib3 1.26.18
[gallery-dl][debug] Configuration Files ['${HOME}/.gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/necromancer-rev/gallery/'
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/necromancer-rev/gallery/'
[deviantart][debug] Using default API credentials (client-id 5388)
[deviantart][info] Requesting public access token
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "POST /oauth2/token HTTP/1.1" 200 128
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/necromancer-rev HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 1s delay.
[deviantart][debug] Sleeping 1.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/necromancer-rev HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 2s delay.
[deviantart][debug] Sleeping 2.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/necromancer-rev HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 3s delay.
[deviantart][info] Register your own OAuth application and use its credentials to prevent this error: https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#extractordeviantartclient-id--client-secret
[deviantart][debug] Sleeping 3.00 seconds (api)

Then I tried using my own Deviant Art account and setting up the refresh-token (I've obfuscated my client-id and client-secret):

{
    "extractor":    {   
        "deviantart":   {
            "skip": true,
            "client-id":        NNNNN,
            "client-secret":    "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",

            "refresh-token":    "cache",
        }
    }
}

I set up the refresh-token:

bash-4.2$ gallery-dl --clear-cache all
[cache][info] Deleted 0 entries from '/home/surfer/.cache/gallery-dl/cache.sqlite3'
bash-4.2$ gallery-dl oauth:deviantart
[oauth][info] Using custom deviantart client ID (NNNNN)
[oauth][info] Opening URL in Firefox:

https://www.deviantart.com/oauth2/authorize?client_id=NNNNN&response_type=code&state=gallery-dl_deviantart_DJIejQcf&redirect_uri=https%3A%2F%2Fmikf.github.io%2Fgallery-dl%2Foauth-redirect.html&duration=permanent&scope=browse+user.manage

Waiting for response. (Cancel with Ctrl+c)
[oauth][info] Writing 'refresh-token' to cache

Your 'refresh-token' is

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

This value has been cached and will automatically be used.

But still no joy on my run:

[gallery-dl][debug] Version 1.26.8
[gallery-dl][debug] Python 3.6.8 - Linux-4.4.2-x86_64-with-centos-7.9.2009-Core
[gallery-dl][debug] requests 2.27.1 - urllib3 1.26.18
[gallery-dl][debug] Configuration Files ['${HOME}/.gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/necromancer-rev/gallery/'
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/necromancer-rev/gallery/'
[deviantart][debug] Using custom API credentials (client-id NNNNN)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/necromancer-rev HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 1s delay.
[deviantart][debug] Sleeping 1.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/necromancer-rev HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 2s delay.
[deviantart][debug] Sleeping 2.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/necromancer-rev HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 3s delay.
[deviantart][debug] Sleeping 3.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/necromancer-rev HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 4s delay.
[deviantart][debug] Sleeping 4.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/necromancer-rev HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 5s delay.
[deviantart][debug] Sleeping 5.00 seconds (api)

The most recent open issue about this Deviant Art '429 Too Many Requests' error is issue #5558, wherein it was noted that DA have significantly lowered their API limits. But I can't download anything at all. I'd think that DA might have 'blacklisted' my IP address but my manual browser downloads work fine.

Thank you for any help.

mikf commented 2 months ago 
bash-4.2$ gallery-dl --clear-cache all

all is not the same as ALL in this context. This did not delete the old access_token and therefore reused it in your next DA download attempt. It should work without 429 errors now, as access tokens are only valid for 60 minutes.