During gallery/folder ripping the DeviantArt avatar fetching fails with 404 due to an old hardcoded address

[rautamiekka]

NOTE: Syntax as understood as of 2017-06-18 at https://github.com/4pr0n/ripme/issues/577#issuecomment-309268167

Gave it a while to see if it would just go away as an error on DA's end, but since it wouldn't I checked the site's source code which confirmed the avatar address is a little different from what's hardcoded into RipMe:

Old=http://a.deviantart.net/avatars/o/USERNAME.gif?11 New=http://a.deviantart.net/avatars/l/o/USERNAME.gif?11

My guess is the trailing ?11 is some sorta spot-specific instruction for the GIF, in which case dropping that part would be in our best interest.

[metaprime]

@rautamiekka I'm getting 404 on your "New" link, even when I change USERNAME to an actual username. Can you provide a set of links that demonstrate this problem and the link you were trying to rip when it occurred?

[rautamiekka]

You're right, even the new link is wrong, and now when looked at it again I see the problem: the leading http://a.deviantart.net/avatars/ part is static and the rest is completely predictable but depends on the username.

For example, suppose your name is

• higglytownhero => http://a.deviantart.net/avatars/h/i/higglytownhero.gif?6
• rautamiekka => http://a.deviantart.net/avatars/r/a/rautamiekka.gif?5

It takes your first letter for the first folder, then the second letter for the second folder, and your whole username with the .gif extension for the filename. In the case of higglytownhero his name is capitalized as HigglyTownHero, so the usernames in links are always lower-case but the case matters in only filenames.

Notice that the trailing integer parts are different from the links in the OP, and quickly checking around (opening up many avatars in succession) it appears the integer is an avatar type indicator since the integer is same across users with similar avatars.

[metaprime]

appears the integer is an avatar type indicator since the integer is same across users with similar avatars.

Is this integer also predictable? Perhaps it's the user account type (indicated on the site with a symbol before the name like ~ or =)?

[rautamiekka]

^ No, it's not user account type indicator and it should be safe to say it has no ties to it either.

For example:

• I'm a normal user, never had Core or its previous equivalent of paid membership, never been a beta tester or anything else other than a normal user, not even banned, suspended or anything and my avatar's (or lack of an avatar) integer is 5, like above, and my symbol is ~. Now when I checked, every normal user has that symbol, since 3 other normal users (http://magicdarkart.deviantart.com/ http://chroniqlo.deviantart.com/ http://dan232323.deviantart.com/) have the same symbol.
• http://jolliapplegirl.deviantart.com/ and http://flareheartmz.deviantart.com/ both have the same beta tester logo, symbol =, next to their username but their avatars have very different integers: http://a.deviantart.net/avatars/j/o/jolliapplegirl.png?2 http://a.deviantart.net/avatars/f/l/flareheartmz.png?6
• http://magnaluna.deviantart.com/ http://newyorkx3.deviantart.com/ http://helmie-d.deviantart.com/ are all Core Members (paid membership), symbol *, but their avatars are http://a.deviantart.net/avatars/m/a/magnaluna.png?14 http://a.deviantart.net/avatars/n/e/newyorkx3.png?4 http://a.deviantart.net/avatars/h/e/helmie-d.jpg?4. You can see large similarity in the 2 latter one's avatar since they're the same integer while magnaluna's seems similar, but not so much.

[rautamiekka]

Sitrep: I couldn't get any versions to spontaneously repeat the avatar fetch failure, but just very recently after literally a couple dozen folders, 1.5.2 started to fail (only the avatar fetching) and these are what I have left since RipMe GUI repeatedly totally froze after being left minimized for extended periods (which is a problem with the particular computer I were using before reinstalling my main one):

Downloading http://a.deviantart.net/avatars/l/mlp-and-anime-rock.jpg?12
http://a.deviantart.net/avatars/l/mlp-and-anime-rock.jpg?12 : Non-retriable status code 404 while downloading http://a.deviantart.net/avatars/l/mlp-and-anime-rock.jpg?12
Downloading http://a.deviantart.net/avatars/n/andoanimalia.png?3
http://a.deviantart.net/avatars/n/andoanimalia.png?3 : Non-retriable status code 404 while downloading http://a.deviantart.net/avatars/n/andoanimalia.png?3
Downloading http://a.deviantart.net/avatars/h/thepossumface.png
http://a.deviantart.net/avatars/h/thepossumface.png : Non-retriable status code 404 while downloading http://a.deviantart.net/avatars/h/thepossumface.png
Downloading http://a.deviantart.net/avatars/a/karpy96.png?3
http://a.deviantart.net/avatars/a/karpy96.png?3 : Non-retriable status code 404 while downloading http://a.deviantart.net/avatars/a/karpy96.png?3

It uses the username's 2nd letter for the first subfolder, doesn't use the 2nd subfolder at all, and uses JPG, PNG or GIF (not seen in this output) for the format. This output is the first when I see RipMe using other than GIF for the avatar file format.

[rautamiekka]

This thing gets more complicated:

Indeed, there are users with an username of a single character, like

• http://b.deviantart.com/ => http://a.deviantart.net/avatars/b/b.jpg?1
• 2-character username like http://bb.deviantart.com/ => http://a.deviantart.net/avatars/b/b/bb.gif.

So for some reason at least 1 user, who's been offline for 300 weeks, still has an JPG avatar with a trailing integer while the other has a GIF without the integer but offline period is a secret, but it's obvious he's been offline for really long too.

So when your username is only 1 character long, DeviantArt uses that one character for the folder and the username with an extension for the filename. Otherwise same but the first 2 characters of the username for the 2 folders.

There are other users' avatars on almost every page, so using a RegEx to catch just the target user's is impossible, therefore it's best to construct the link in app.