search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

[deviantart] jwt token validation failed #4548

Closed mdashlw closed 1 year ago

mdashlw commented 1 year ago

DeviantArt extractor works by faking a JWT token by using alg=none, it appears to be no longer working.

Unsigned alg=none: https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0. (returns 401 token validation failed) SIgned alg=HS256 by deviantart: https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.kNkEHMmmwEAdVlKIco8-ZZkxuttRF2KT39Yot4vhXa8 (works)

JWT payload for both of these is exactly the same:

{
  "sub": "urn:app:7e0d188982264373a5f0d415ea0d26e0",
  "iss": "urn:app:7e0d188982264373a5f0d415ea0d26e0",
  "obj": [
    [
      {
        "path": "/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png"
      }
    ]
  ],
  "aud": [
    "urn:service:file.download"
  ]
}
ClosedPort22 commented 1 year ago

That sucks...

Polygon1177 commented 1 year ago

I got those errors with Deviantart as well, since yesterday. Wasn't an issue till the day before.

Mirrorman95 commented 1 year ago

I recently reported a similar set of errors befalling another downloader.

Twi-Hard commented 1 year ago

I wrote this in a different issue as an edit to my existing post so I don't know if anybody even saw it. I just wanted to mention downloading the full res of this paid art still works. I don't know if that's significant or might help or not.

https://www.deviantart.com/darkflame75/art/The-Moon-Rises-603249824

mdashlw commented 1 year ago

I wrote this in a different issue as an edit to my existing post so I don't know if anybody even saw it. I just wanted to mention downloading the full res of this paid art still works. I don't know if that's significant or might help or not.

https://www.deviantart.com/darkflame75/art/The-Moon-Rises-603249824

Cannot reproduce. [downloader.http][warning] '401 Unauthorized' for 'https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/bfdad229-dae3-43ef-9a85-11cd8c975e4b/d9z5qjk-5a091042-6cfa-4b47-9022-e000e62fbe09.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mL2JmZGFkMjI5LWRhZTMtNDNlZi05YTg1LTExY2Q4Yzk3NWU0Yi9kOXo1cWprLTVhMDkxMDQyLTZjZmEtNGI0Ny05MDIyLWUwMDBlNjJmYmUwOS5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ.'

Twi-Hard commented 1 year ago

I tested it again right before posting my comment. It must be because I downloaded it right before the issue began so it still lets me I guess.

ClosedPort22 commented 1 year ago

I tested it again right before posting my comment. It must be because I downloaded it right before the issue began so it still lets me I guess.

That sounds very unlikely. Can you access the URL in a browser?

Twi-Hard commented 1 year ago

Yes.

❯ file 'deviantart_603249824_The Moon Rises.png'
deviantart_603249824_The Moon Rises.png: PNG image data, 7680 x 4320, 8-bit/color RGBA, non-interlaced

https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/bfdad229-dae3-43ef-9a85-11cd8c975e4b/d9z5qjk-5a091042-6cfa-4b47-9022-e000e62fbe09.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mL2JmZGFkMjI5LWRhZTMtNDNlZi05YTg1LTExY2Q4Yzk3NWU0Yi9kOXo1cWprLTVhMDkxMDQyLTZjZmEtNGI0Ny05MDIyLWUwMDBlNjJmYmUwOS5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ. The dot is a part of the url. It was downloaded without an archive and with --no-skip

mdashlw commented 1 year ago

Yes.

❯ file 'deviantart_603249824_The Moon Rises.png'
deviantart_603249824_The Moon Rises.png: PNG image data, 7680 x 4320, 8-bit/color RGBA, non-interlaced

https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/bfdad229-dae3-43ef-9a85-11cd8c975e4b/d9z5qjk-5a091042-6cfa-4b47-9022-e000e62fbe09.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mL2JmZGFkMjI5LWRhZTMtNDNlZi05YTg1LTExY2Q4Yzk3NWU0Yi9kOXo1cWprLTVhMDkxMDQyLTZjZmEtNGI0Ny05MDIyLWUwMDBlNjJmYmUwOS5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ. The dot is a part of the url. It was downloaded without an archive and with --no-skip

Can you access the url in incognito? Could you try reproducing with curl?

Twi-Hard commented 1 year ago

It works in incognito. It doesn't work with curl:

❯ cat image.png
token validation failed
ClosedPort22 commented 1 year ago

It could be a regional cache sort of thing. I just checked and Cloudfront caches the images for a long period of time:

cache-control: public, max-age=2592000, immutable
age: 74973
mdashlw commented 1 year ago

It could be a regional cache sort of thing. I just checked and Cloudfront caches the images for a long period of time:

cache-control: public, max-age=2592000, immutable
age: 74973

Likely. you can bypass Cloudfront cache by adding random query params like ?token=...&1234

zakman4466 commented 1 year ago

To confirm, once the commit goes live, should the issue with these errors (and the associated 401 errors) be resolved?

kattjevfel commented 1 year ago

@zakman4466 it "resolves" it by just always going with the lowres images since you can't get the highres pictures anyway.

mikf commented 1 year ago

Ironchest337 did it again! https://github.com/mikf/gallery-dl/commit/20d1683c47af81f024b8d8bc09894f54e2a393b3

ClosedPort22 commented 1 year ago

Ironchest337 did it again! 20d1683

Weirdly enough, that trick does not work for the example given by @mdashlw (403 Forbidden), and that's the only exception I've noticed so far. At first I thought it was because it was a relatively new submission, but later I was able to download even newer submissions.

mikf commented 1 year ago

It works when using the full DA URL, but this post is also downloadable without any JWT shenanigans. It doesn't actually get touched by any JWT logic to begin with, even when original is disabled.

Ironchest337 commented 1 year ago

Weirdly enough, that trick does not work for the example given by @mdashlw (403 Forbidden), and that's the only exception I've noticed so far. At first I thought it was because it was a relatively new submission, but later I was able to download even newer submissions.

I went ahead and checked and I believe it's an issue of the original token they are using. Removing everything after urn:app: for sub and iss allows it to work

mdashlw commented 1 year ago

It seems to accept iss=urn:app:7e0d188982264373a5f0d415ea0d26e0 only with signed tokens, e.g. https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.kNkEHMmmwEAdVlKIco8-ZZkxuttRF2KT39Yot4vhXa8 does work. It accepts any sub though. Weird but very glad it's working again.

Mirrorman95 commented 1 year ago

It seems to accept iss=urn:app:7e0d188982264373a5f0d415ea0d26e0 only with signed tokens, e.g. https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.kNkEHMmmwEAdVlKIco8-ZZkxuttRF2KT39Yot4vhXa8 does work. It accepts any sub though. Weird but very glad it's working again.

Just to be clear, what gallery-dl feature exactly is working again?

mdashlw commented 1 year ago

It seems to accept iss=urn:app:7e0d188982264373a5f0d415ea0d26e0 only with signed tokens, e.g. https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.kNkEHMmmwEAdVlKIco8-ZZkxuttRF2KT39Yot4vhXa8 does work. It accepts any sub though. Weird but very glad it's working again.

Just to be clear, what gallery-dl feature exactly is working again?

Same as before, i.e. downloading all deviantart images, including paid, in original resolution

Mirrorman95 commented 1 year ago

It seems to accept iss=urn:app:7e0d188982264373a5f0d415ea0d26e0 only with signed tokens, e.g. https://wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/1bf37f96-f0c0-4978-87d5-fec7a7860e96/dg9ewam-1771d013-9bb1-4bc9-82f7-3d145dbb4c76.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcLzFiZjM3Zjk2LWYwYzAtNDk3OC04N2Q1LWZlYzdhNzg2MGU5NlwvZGc5ZXdhbS0xNzcxZDAxMy05YmIxLTRiYzktODJmNy0zZDE0NWRiYjRjNzYucG5nIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.kNkEHMmmwEAdVlKIco8-ZZkxuttRF2KT39Yot4vhXa8 does work. It accepts any sub though. Weird but very glad it's working again.

Just to be clear, what gallery-dl feature exactly is working again?

Same as before, i.e. downloading all deviantart images, including paid, in original resolution

I just tried that The Moon Rises example (using https://github.com/mhogomchungu/media-downloader as a GUI), but it gave me this error:

/gallery-dl" "https://www.deviantart.com/darkflame75/art/The-Moon-Rises-603249824" [downloader.http][warning] '401 Unauthorized' for 'https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/bfdad229-dae3-43ef-9a85-11cd8c975e4b/d9z5qjk-5a091042-6cfa-4b47-9022-e000e62fbe09.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mL2JmZGFkMjI5LWRhZTMtNDNlZi05YTg1LTExY2Q4Yzk3NWU0Yi9kOXo1cWprLTVhMDkxMDQyLTZjZmEtNGI0Ny05MDIyLWUwMDBlNjJmYmUwOS5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ.' [download][info] Trying fallback URL #1

Then it downloaded the low-resolution preview image. I went to the URL and it said "token validation failed". Another deviation I tried also insisted on using a low-res fallback. If original-res downloading is working again, then what am I doing wrong?

mdashlw commented 1 year ago

Then it downloaded the low-resolution preview image. I went to the URL and it said "token validation failed". Another deviation I tried also insisted on using a low-res fallback. If original-res downloading is working again, then what am I doing wrong?

the fix isn't released yet. install latest gallery-dl directly from github:

python3 -m pip install -U -I --no-deps --no-cache-dir https://github.com/mikf/gallery-dl/archive/master.tar.gz
Mirrorman95 commented 1 year ago

Then it downloaded the low-resolution preview image. I went to the URL and it said "token validation failed". Another deviation I tried also insisted on using a low-res fallback. If original-res downloading is working again, then what am I doing wrong?

the fix isn't released yet. install latest gallery-dl directly from github:

python3 -m pip install -U -I --no-deps --no-cache-dir https://github.com/mikf/gallery-dl/archive/master.tar.gz

That did it. I have to run it from my Python3 folder on the command line, but it works. Thank you.

EDIT: Some of the deviations in three specific galleries I tried still give me errors that include " (no refresh-token)". Other people's galleries give me no such trouble at all.

EDIT2: Here are links to a few of those images. They're not even blurred and don't even have pay buttons, but they are NSFW: https://www.deviantart.com/fenyxstrikerart/art/The-Lammiath-Mask-nude-Photography-910498134 https://www.deviantart.com/fenyxstrikerart/art/The-Lammiath-Mask-2-nude-Photography-910500549

The errors they give me are: gallery-dl https://www.deviantart.com/fenyxstrikerart/art/The-Lammiath-Mask-nude-Photography-910498134 [deviantart][info] Requesting public access token [deviantart][warning] Unable to access premium content (no refresh-token) [deviantart][info] No results for https://www.deviantart.com/fenyxstrikerart/art/The-Lammiath-Mask-nude-Photography-910498134