deviantart rip went from taking 20-30minutes in 2022 to 8-10hours in 2023. Could public private token toggle after seeing a 400 error?

[left1000]

deviantart rip went from taking 20-30minutes in 2022 to 8-10hours in 2023. Could public private token toggle after seeing a 400 error?

So the gist is, 1.25.2 updated things to fix the 400 errors, by using private api tokens instead of public api tokens, but this is slow.

Most galleries that have paid content in them are still 90% free content. Would there be a way to code things so that gallery-dl would run with the faster public token, then after getting a 400 error re-run that one url via the private api token?

Repeating 400 errors in this manner would likely reduce 429 errors by 90%? I'd reckon?

Unless I entirely misunderstand the problem? https://github.com/mikf/gallery-dl/issues/3845

Note: I made this a new issue because this is now a feature request not a debugging question. I can't seem to figure out how to add the feature request tag though.

[cheese529]

i've been getting tons of 429 too many requests errors that go up to a using 15 second delay, what exactly does this mean? are those pictures not being downloaded and then skipped or are they still eventually downloaded after the 429 error hits 15 seconds and expires ?

[left1000]

i've been getting tons of 429 too many requests errors that go up to a using 15 second delay, what exactly does this mean? are those pictures not being downloaded and then skipped or are they still eventually downloaded after the 429 error hits 15 seconds and expires ?

429 means the program is pausing and will retry to connect in the near future, but after 15 seconds it might just come back and tell you it wants to wait for another 16 seconds....

Which is how a 30minute task became a 10hour task, although it's deviantart's fault, I'm just wondering if there might be anyway for gallery-dl to better mitigate it.

[cheese529]

ahh I see. i have been getting these 2 errors as well. do either of these means that it is failing to download certain content or pictures? [2023-04-21T10:56:13][warning] API responded with 429 Too Many Requests. Using 15s delay. [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T10:59:01][warning] HTTPSConnectionPool(host='images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com', port=443): Read timed out. (read timeout=30.0) (1/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T10:59:35][warning] HTTPSConnectionPool(host='images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com', port=443): Read timed out. (2/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T11:00:10][warning] HTTPSConnectionPool(host='images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com', port=443): Read timed out. (3/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T11:00:43][warning] HTTPSConnectionPool(host='images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com', port=443): Read timed out. (4/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T11:01:17][warning] HTTPSConnectionPool(host='images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com', port=443): Read timed out. (read timeout=30.0) (5/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T11:01:17][error] Failed to download black_widow_booty_by_inspector97-ddcbc1f.jpg [Source URL: https://www.deviantart.com/223s/favourites]

[2023-04-21T09:49:06][warning] API responded with 429 Too Many Requests. Using 15s delay. [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T09:50:22][warning] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (1/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T09:50:49][warning] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (1/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T09:51:09][warning] API responded with 429 Too Many Requests. Using 4s delay. [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T09:51:13][warning] API responded with 429 Too Many Requests. Using 5s delay. [Source URL: https://www.deviantart.com/223s/favourites]

[left1000]

ahh I see. i have been getting these 2 errors as well. do either of these means that it is failing to download certain content or pictures? [2023-04-21T10:56:13][warning] API responded with 429 Too Many Requests. Using 15s delay. [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T10:59:01][warning] HTTPSConnectionPool(host='images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com', port=443): Read timed out. (read timeout=30.0) (1/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T10:59:35][warning] HTTPSConnectionPool(host='images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com', port=443): Read timed out. (2/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T11:00:10][warning] HTTPSConnectionPool(host='images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com', port=443): Read timed out. (3/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T11:00:43][warning] HTTPSConnectionPool(host='images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com', port=443): Read timed out. (4/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T11:01:17][warning] HTTPSConnectionPool(host='images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com', port=443): Read timed out. (read timeout=30.0) (5/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T11:01:17][error] Failed to download black_widow_booty_by_inspector97-ddcbc1f.jpg [Source URL: https://www.deviantart.com/223s/favourites]

[2023-04-21T09:49:06][warning] API responded with 429 Too Many Requests. Using 15s delay. [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T09:50:22][warning] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (1/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T09:50:49][warning] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (1/5) [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T09:51:09][warning] API responded with 429 Too Many Requests. Using 4s delay. [Source URL: https://www.deviantart.com/223s/favourites] [2023-04-21T09:51:13][warning] API responded with 429 Too Many Requests. Using 5s delay. [Source URL: https://www.deviantart.com/223s/favourites]

What two errors? That log looks like just 429 error, multiple times. Which is the intended function of the program, and doesn't mean anything bad. Although now that I think of it the 5/5 might mean that you've gotten so many 429 errors it's decided to skip a file. I'm not sure. Someone with more experience will have to chime in, and also, if true, there should be a .conf setting to change the limit from 5 to a bigger number like 10, and if there is an option to do that, I suppose I'd want to make the change as well.

Although this issue is totally separate from the one I described in the opening post, and probably should be it's own separate issue, and since it's not a separate issue, it's less likely to get an answer.

[left1000]

Anyone know how I get an issue labeled a feature request?

[GotoConsol]

Besides of course Oauth credentials in config file

If you provide username and password, practically solves the problem

gallery-dl -u YourUsername -p YourPassword deviantart.com/author001

or put them into config file

[cloudywings2]

Besides of course Oauth credentials in config file

If you provide username and password, practically solves the problem

gallery-dl -u YourUsername -p YourPassword deviantart.com/author001

or put them into config file

Hello! I just wanted to let you know that this doesn't work. My config file has the following:

"cookies": "cookies.txt", "cookies-update": true, "username": "##username", "password": "##password", "client-id": "##clientID", "client-secret": "##clientSecret", "refresh-token": "##refreshToken",

and I'm still getting the same error. Nearly every file Gallery-DL attempts to download from DA is accompanied by a 15-30 second pause and connection errors.

[GotoConsol]

I see, then check the following:

remove username and pass from config

and try this:

gallery-dl -u YourUsername -p YourPassword deviantart.com/author001

[cloudywings2]

I see, then check the following:

remove username and pass from config

and try this:

gallery-dl -u YourUsername -p YourPassword deviantart.com/author001

So this does work! However, normally I download all my galleries from a text file link list. Is there a way I can change my configuration settings to mimic the behavior of the command you gave me? What is the issue that is causing this?

[GotoConsol]

In fact I don't know what is the issue behind this phenomenon and I also don't know how to mimic this behavior via config file Let us simply ask @mikf

[mikf]

Should be "fixed" in commit https://github.com/mikf/gallery-dl/commit/98c9fdb4145142280280b7320a17ec9c6ef02742, although this can and will be improved further.

---

So this does work!

Username and password have no effect except for stash content and searches.

In fact I don't know what is the issue behind this phenomenon and I also don't know how to mimic this behavior via config file

Try to download from anything that makes gallery-dl use a private access token for API requests, like https://www.deviantart.com/watch/deviations, and you'll be in 429 hell soon enough.

[cloudywings2]

My apologies - I was actually wrong in my prior post; I tried it again with other links and it gave the same issue. So to clarify, we can resolve this issue by updating the program?

What would be an example of a deviation that uses a private access token? Would this include artists with Watchers Only deviations or those you subscribe to? If so, how can we prevent "429 hell?"

[cheese529]

yep this is true, also any NSFW deviations will put you into 429 hell as well.

[cheese529]

@mikf here is my current config for deviantart, is there anything that you would recommend me to change in order to help prevent those errors as well? should i add username and password in the config or run it with the command? }, "deviantart": { "filename": "{filename}.{extension}", "client-id": "....", "client-secret": ".....", "refresh-token": ".....", "auto-watch": true, "auto-unwatch": true, "comments": false, "extra": true, "flat": true, "folders":false, "group": true, "include": "all", "journals": "html", "mature": true, "metadata": true, "original": true, "wait-min": 0

[left1000]

My apologies - I was actually wrong in my prior post; I tried it again with other links and it gave the same issue. So to clarify, we can resolve this issue by updating the program?

What would be an example of a deviation that uses a private access token? Would this include artists with Watchers Only deviations or those you subscribe to? If so, how can we prevent "429 hell?"

This is why I was trying to say that my issue was a feature request, the problem you're having is caused by deviantart, and no setting fixes it. 429 errors are the intended behavior to deal with a problem deviantart caused. It won't be "fixed" until gallery-dl is updated with a fix. Also 429 isn't really an "error" in the sense that nothing is wrong. Deviantart is telling the software that you've made too many requests in too short a time, please wait.

[left1000]

Looks like it's working great again! thanks https://github.com/mikf/gallery-dl/releases/tag/v1.25.3