Deviantart Cookies & 500 Internal Server Errors

[DallasWhite]

So I keep getting this output for every deviantart profile I try to scrape:

gallery-dl https://www.deviantart.com/leatherruffian
[deviantart][error] HttpError: '500 Internal Server Error' for 'https://www.deviantart.com/api/v1/oauth2/user/profile/leatherruffian'
[deviantart][warning] No session cookies set: Disabling original file downloads.

Here's what I have in my config file:

}
    "extractor":
    {
        "base-directory": "F:/Scripts/Gallery-dl/",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
        "archive": "F:/Scripts/Gallery-dl/archive.sqlite3",

        "postprocessors": [
                  { 
               "name": "ugoira",
               "whitelist": ["pixiv", "danbooru"],
                       "ffmpeg-twopass": true,
                       "ffmpeg-args": ["-c:v", "libvpx", "-crf", "4", "-b:v", "5000k", "-an"],
               "ffmpeg-location": "C:/Users/ABlack/gallery-dl/ffmpeg.exe"
                  }
                  ],

             "deviantart":
                   {
                 "username": "ShakeDown89",
                         "password": "######",
                 "client-id": "######",
                 "client-secret": "######",
                 "refresh-token": "######",
                 "include": "gallery,scraps"
              },
...

It used to work fine before, but I'm not sure what I could have changed. Also, I didn't need a session cookie to get the original file either, but it's asking for one now. Any assistance would be great.

[biznizz]

It's probably for the best that you add in a link to a cookies.txt file exported from your browser. This is my set up, for example:

"deviantart":
        {
            "include": "gallery,scraps",
            "refresh-token": "SECRET",
            "client-id": "SECRET",
            "client-secret": "SECRET",
            "flat": true,
            "folders": false,
            "journals": "html",
            "mature": true,
            "metadata": false,
            "original": true,
            "quality": 100,
            "extra": true,
            "wait-min": 0,
            "cookies": "C:\\Users\\USER\\cookies.txt",
            "cookies-update": true

        },

[DallasWhite]

I tried using the cookies.txt from chrome, but I get a 404 error:

[deviantart][error] HttpError: '404 Not Found' for 'https://www.deviantart.com/_napi/da-user-profile/api/gallery/contents'

[biznizz]

What I did was, I didn't use the original cookies.txt, I copied the relevant DA cookie info and made a separate one and pointed my config to it.

Some of the cookies start with www.deviantart or #HttpOnly_.deviantart or stuff like that, the program usually only likes it when it just starts off with .deviantart. So copy it over and make all the lines start with '.deviantart`.

Again, here's my example for my DA cookies.txt:

# Netscape HTTP Cookie File

.deviantart.com TRUE    /   FALSE   1648357312  _ga GA###
.deviantart.com TRUE    /   FALSE   1585285349  _gat    1
.deviantart.com TRUE    /   FALSE   1585285349  _gat_ThirdPartyTracking_UA_###
.deviantart.com TRUE    /   FALSE   1585371712  _gid    GA###
.deviantart.com TRUE    /   FALSE   1587877225  auth    __###
.deviantart.com TRUE    /   TRUE    2145945601  auth_secure __###
.deviantart.com TRUE    /   FALSE   0   td  ###
.deviantart.com TRUE    /   FALSE   1587841701  userinfo    __###
.deviantart.com TRUE    /   FALSE   1587877722  vd  __###
.deviantart.com TRUE    /_sisu/do/  FALSE   1616785700  _pxhd       ###
.deviantart.com TRUE    /users/ FALSE   1616785689  _pxhd   ###

[DallasWhite]

Yeah I had already done that to copy the netscape format, same deal.

[biznizz]

Hm, when was the last time you got a new Oauth refresh token?

[DallasWhite]

The second I started having the issue today.

[biznizz]

And the cookies.txt that your config links to, it's fully up-to-date with cookies you got from the Eclipse version of DA?

Sometimes, the cookies-refresh can cause certain data points to update incorrectly, like your DA username not showing up in your userinfo or some field like auth or auth_secure to disappear.

My only suggestions are to clear all your DA cookies from your browser, log in with DA Eclipse, interact with the site some (go to different pages, etc) to get new cookies, export them, copy all the relevant cookies to the separate cookies file your config is linked to, refresh oauth and update config, and make sure that gallery-dl is the latest version, including possibly using a dev build if you need to.

After looking up what a 500 Internal Server Error is for DA, it sounds like whatever server you're connecting to is having issues, so I can't really help out with that, other than also suggesting to see if you could refresh your ISP address.

[DallasWhite]

Oh interesting, I hadn't considered the Eclipse version. I still use the classic version.

[biznizz]

Yeah, I think the cookies you get from the Eclipse version are the ones that work best with gallery.

Have you had any more issues?

[DallasWhite]

That did the trick! Looks like gallery-dl has been written to handle the Eclipse version.

[biznizz]

Yeah, the DA devs has pretty much abandoned Classic DA in favor of Eclipse, and making gallery-dl work properly requires Eclipse now.

I'm still not super thrilled I have to use Eclipse from now on (I do still prefer Classic), but it is what it is, to get this useful program to work.