[deviantart] API responded with 429 Too Many Requests constantly, --sleep-request 5.0-10.10 and even 30.0 doesn't help

[unmnae]

About day or two ago i had been downloading galleries from deviantart. Usually it goes for a few pictures, then starts to increase delays up to 30 seconds, then starts to download one picture per ~ 3 to 7 429 errors, which were slow, but fine enough for me. This way it had worked for a year at least. At some moment i had noticed, that 429 error messages had started to go without messages of a picture being downloaded, right during download. Since that time it now doesn't download anything but keeps returning 429 errors. I had tried to go back to 12.6.5 of gallerydl, but it didn't help. Site itself opens normally, pictures are downloaded normally through browser. Could it be that deviantart had started to filter out requests from gallerydl specifically?

[Hrxn]

Are you using your own OAuth credentials? I can't tell, you did not provide a verbose log.

[GoAwayNow]

Can confirm, two days ago Deviantart started responding with 429 to any request I made. This was tested with credentials that came from my already-logged-in instance, as well as with credentials spawned by private browsing instances and entirely new browsers I don't use.

I'm not using my own OAuth application.

I'm running through Hydownloader, so I don't know how to get a verbose log, but this is a (mildly redacted) log produced by it.

single-urls-6685-gallery-dl-latest - Copy.txt

EDIT: I've registered my own application and used it instead, which worked. I think this implies that the issue is exactly what it says, the default OAuth application is over its limits.

Follow the instructions in the configuration docs to set up your own application. BTW, this link is to the raw file because Github fails to properly render large files. Search for "extractor.deviantart.client-id" to find what you're looking for.

[unmnae]

Follow the instructions in the configuration docs to set up your own application. BTW, this link is to the raw file because Github fails to properly render large files. Search for "extractor.deviantart.client-id" to find what you're looking for.

Yes, this seems to fix my problem. But it is still looks like a strange change of deviantart's side.

UPD: now it seem to download .jpg files instead of full-sized .png in some cases, even when there is "download" button, that manually allows to download original png through browser.

UPD2: had seen several config options like "quality": png, but then it starts to fail to download pictures without "download" button on their page, that is available only in jpg. And links, that download jpg instead of png when "download" button is present on page and leads to png, still download jpg. As if he ignores download button on some of them.

Here is my config

{ "extractor": { "deviantart": { "#": "download 'gallery' and 'scraps' images for user profile URLs", "include": "gallery,scraps",

    "mature": true,
    "original": true,
        "quality": "png",
    "public": false,

        "#": "put description texts into a separate directory",
        "metadata": true,
        "postprocessors": [
            {
                "name": "metadata",
                "mode": "custom",
                "directory"       : "Descriptions",
                "content-format"  : "{description}\n",
                "extension-format": "descr.txt"
            }
          ],
        "cookies": "path_to_cookie_file/cookies-deviantart-com.txt",
        "cookies-update": true,
        "client-id": MY_CORRENT_CLIEND_ID,
        "client-secret": "MY_CLIENT_SECRET"
        }
}

}

Here are commands i use: gallery-dl.exe -i my_path_to_my_own_deviantart_links_input_file\input.txt --config my_path_to_config\gallerydl\config.json -o public=0 --sleep-request 5.0-10.0 --dest path_to_destination --download-archive my_path_to_archive\archive-DA.sqlite3

gallery-dl.exe --config my_path_to_config\config.json -o public=0 --dest path_to_destination link_to_page_with_download_button_and_png_picture

[Hrxn]

EDIT: I've registered my own application and used it instead, which worked. I think this implies that the issue is exactly what it says, the default OAuth application is over its limits.

Yes, that's exactly the case.

Follow the instructions in the configuration docs to set up your own application. BTW, this link is to the raw file because Github fails to properly render large files. Search for "extractor.deviantart.client-id" to find what you're looking for.

Rendering of rst seems still somehwat broken on GitHub.

Here's the same content with proper formatting etc: https://gdl-org.github.io/docs/configuration.html#api-tokens-ids

[GoAwayNow]

Hey, turns out, this isn't just an "Oopsy, default app got used too much!"

Deviantart seems to have SIGNIFICANTLY lowered their API limits. My new app, made two days ago, is already returning 429.

EDIT: And based on their supposed "adaptive rate limiting" and the fact that the default app has consistently returned 429 for days, it does seem plausible that Deviantart is attempting to block Gallery-DL use.

[mikf]

UPD: now it seem to download .jpg files instead of full-sized .png in some cases, even when there is "download" button, that manually allows to download original png through browser.

You probably need a refresh-token. (gallery-dl oauth:deviantart)

[GoAwayNow]

You probably need a refresh-token. (gallery-dl oauth:deviantart)

I thought refresh tokens were stored in cache.sqlite3. Do they need to be copied into the config file as well?

[unmnae]

gallery-dl oauth:deviantart

That i do before each time i start new download, just in case. It is there is some pictures, whose previews can be expanded (when viewing through browser). Some of this expandable pictures can be expanded once (and disable RMB context menu), some can be expanded twice - not disabling this menu. And for this pictures, that can be expanded twice AND which are jpg in preview, but have download button, which will downloads png, gallerydl NOW had started to download jpg expanded previews. Problem is that it doesn't work like that for all this types of pictures, but for some of them. As soon as i will find an sfw example, i will add it to this post. It wasn't an issue before i had started to receieve 429 constantly and were forced to use this

Follow the instructions in the configuration docs to set up your own application.

solution to download anything at all. Before that login/password in config file and oauth command had worked nicely for me.

UPD1: Btw, could it be connected to sfw/nsfw setting? Because it seem like in gallery with some images marked by nswf (when logged out) specifically this images get their jpeg preview being downloaded instead of png original despite availability of a download button when logged in and png behing this download button despite "mature" being enabled both in account on site and have "mature": true in my config.

[Brad3273]

Deviantart seems to have SIGNIFICANTLY lowered their API limits. My new app, made two days ago, is already returning 429.

Is it at all possible to download much from Deviant Art these days?

I'm using gallery-dl 1.26.8. With my .gallery-dl.conf set up as always with my client-id and client-secret, and with a refreshed refresh-token, a dlownload of a single Deviant Art gallery works fine for a while until I get 50 or 100 images, and then I start getting 429 'Too Many Requests' and things slow to an impractical crawl.

I wondered if using the 'public' token would be any better, so I removed the client and refresh tokens from the configuration file, put in 'public: true' ... but gallery-dl saw 429 errors immediately:

@[gallery-dl][debug] Version 1.26.8 [gallery-dl][debug] Python 3.6.8 - Linux-4.4.2-x86_64-with-centos-7.9.2009-Core [gallery-dl][debug] requests 2.27.1 - urllib3 1.26.18 [gallery-dl][debug] Configuration Files ['${HOME}/.gallery-dl.conf'] [gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/necromancer-rev/gallery/' [deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/necromancer-rev/gallery/' [deviantart][debug] Using default API credentials (client-id 5388) [deviantart][info] Requesting public access token [urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443 [urllib3.connectionpool][debug] https://www.deviantart.com:443 "POST /oauth2/token HTTP/1.1" 200 128 [urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/necromancer-rev HTTP/1.1" 429 105 [deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"} [deviantart][warning] API responded with 429 Too Many Requests. Using 1s delay. [deviantart][debug] Sleeping 1.00 seconds (api) [urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/necromancer-rev HTTP/1.1" 429 105 [deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"} [deviantart][warning] API responded with 429 Too Many Requests. Using 2s delay. [deviantart][debug] Sleeping 2.00 seconds (api) ...

Is the public access unusable with Deviant Art? Are they limiting me on IP address?

How are people downloading anything beyond small quantities from Deviant Art these days? Thanks!