[Deviantart] Non Stop 429 Errors and 403 Forbidden Errors

[cheese529]

Hey there, hope everyone is doing well. My apologies in advance for the spam of questions but it's been around a year since i've used gallery-dl to download from Deviant Art and i'm guessing a lot has changed since then.

After just downloading about 30 images I start getting 429 errors and then on some images I get a 403 forbidden error where gallery-dl does a fallback and downloads a lower quality version of the image. (NOTE: the image giving 403 forbidden error has downloads enabled upon checking via browser)

Now regarding the images that have downloads disabled. I did some quick reading and apparently as of right now there is no work around to download images that have disabled downloads in full resolution if I am correct?

Here is my current config for DA. https://gist.github.com/cheese529/a9beaa310611465e9b9757f27e65f6d9 I believe I have it setup correctly with client id and refresh token but if anyone can tell me if something is wrong or if this is just an issue with DA i'd greatly appreciate it.

[growingProvince]

Just want to add to this that since a few days I too have been receiving 403 errors with images that have a download option in the deviantart interface. I also have an account and token configured that have worked correctly until recently.

[Ahmed0425]

Duplicate #4652

[cheese529]

Duplicate #4652

not a duplicate, that was for images that have downloads disabled.

[albert-softie]

i've been having the same problem, deviantart downloads will work fine for a while before eventually getting hit with back to back 429 errors until finally settling on a 21 second delay, regardless of whether images have downloads enabled or not here's my config setup for DA, which used to work totally fine but for some reason now doesn't:

    "extractor":
    {
        "base-directory": "[REDACTED]",
        "parent-directory": false,
        "postprocessors": null,
        "archive": null,
        "cookies": null,
        "cookies-update": true,
        "proxy": null,
        "skip": true,

        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0",
        "retries": 4,
        "timeout": 30.0,
        "verify": true,
        "fallback": true,

        "sleep": 0,
        "sleep-request": 0,
        "sleep-extractor": 0,

        "path-restrict": "auto",
        "path-replace": "_",
        "path-remove": "\\u0000-\\u001f\\u007f",
        "path-strip": "auto",
        "extension-map": {
            "jpeg": "jpg",
            "jpe" : "jpg",
            "jfif": "jpg",
            "jif" : "jpg",
            "jfi" : "jpg"
        },
        "deviantart":
        {
            "jwt": false,
            "client-id": "[REDACTED]",
            "client-secret": "[REDACTED]",
            "refresh-token": "cache",
            "comments": true,
            "extra": true,
            "flat": true,
            "folders": false,
            "include": ["gallery", "scraps", "journal", "status"],
            "journals": "html",
            "cookies": ["brave"],
            "mature": true,
            "auto-watch": true,
            "metadata": true,
            "original": true,
            "wait-min": 0,
            "filename": "{category}_{index}_{date}_{title}.{extension}",
            "postprocessors": [
                {
                    "name": "metadata",
                    "mode": "custom",
                    "directory"       : "Descriptions",
                    "content-format"  : "{description}\n",
                    "extension-format": "descr.html"
                },
                {
                    "name": "metadata",
                    "mode": "custom",
                    "directory"       : "Descriptions",
                    "content-format"  : "{comments}\n",
                    "extension-format": "comments.html"
                },
                {
                    "name": "metadata",
                    "mode": "custom",
                    "directory"       : "Status",
                    "content-format"  : "{status}\n",
                    "extension-format": "status.html"
                }
            ]
        },

[mikf]

on some images I get a 403 forbidden error where gallery-dl does a fallback and downloads a lower quality version of the image

You are either running an older gallery-dl version or you have jwt enabled. Downloading original files from posts without download button does no longer work and you now need a refresh-token for "mature" posts.

Speaking of, private tokens generated from a refresh token have a much lower rate limit than public tokens and will sooner or later result in 429 errors like albert-softie reports. There is no known way around that. Maybe setting a high-enough sleep-request delay could help.

After just downloading about 30 images I start getting 429 errors

This shouldn't happen that fast. Are you sure gallery-dl is actually using your API credentials? Check the --verbose output.

$ gallery-dl -v deviantart.com/USERNAME
...
[gallery-dl][debug] Configuration Files ['${HOME}/.gallery-dl.conf']
...
[deviantart][debug] Using custom API credentials (client-id 12345)
...
[deviantart][debug] Switching to private access token
[deviantart][info] Refreshing private access token

right now there is no work around to download images that have disabled downloads in full resolution if I am correct?

correct.

[Brad3273]

My attempts to download images from Deviant Art with gallery-dl 1.26.6 are immediately returning with 'API responded with 429 Too Many Requests.' warnings and multi-second sleeps.

Running with '-v' I see the line:

[deviantart][debug] Using custom API credentials (client-id NNNNN)

... but I never see the two lines about using the private access token that has been listed in an earlier comment.

I have 'client-id', 'client-secret' set for extractor.deviantart, and extractor.deviantart.refresh-token is set to 'cache'.

I've run 'gallery-dl --clear-cache all' and then 'gallery-dl oauth:deviantart' and the latter ran successfully.

This is the first few lines of output from 'gallery-dl -v "https://www.deviantart.com/XXXXX/gallery/"' -

[gallery-dl][debug] Version 1.26.6
[gallery-dl][debug] Python 3.6.8 - Linux-4.4.2-x86_64-with-centos-7.9.2009-Core
[gallery-dl][debug] requests 2.27.1 - urllib3 1.26.18
[gallery-dl][debug] Configuration Files ['${HOME}/.gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/XXXXXXXXXXX/gallery/'
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/XXXXXXXXXXX/gallery/'
[deviantart][debug] Using custom API credentials (client-id NNNNNN)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/XXXXXXXXXXX HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 1s delay.
[deviantart][debug] Sleeping 1.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/XXXXXXXXXXX HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 2s delay.
[deviantart][debug] Sleeping 2.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/XXXXXXXXXXX HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 3s delay.
[deviantart][debug] Sleeping 3.00 seconds (api)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/XXXXXXXXXXX HTTP/1.1" 200 331
[deviantart][debug] Sleeping 2.00 seconds (api)

Is there something else I need to do to get the refresh-token magic working? Thanks.

[rautamiekka]

This is the first few lines of output from 'gallery-dl -v "https://www.deviantart.com/XXXXX/gallery/"' -

Don't censor the link, it only makes things harder.

[mikf]

@Brad3273 Try it again. If my guess is correct, it should have "fixed" itself by now, as the old access_token from the default client has expired.

[Brad3273]

@Brad3273 Try it again. If my guess is correct, it should have "fixed" itself by now, as the old access_token from the default client has expired.

You're right! It ran flawlessly today. Thank you.