search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

Deviantart Extractor - Image Resolution Issue #293

Closed biznizz closed 3 years ago

biznizz commented 5 years ago

Have just noticed a strange quirk using Deviantart extractor; while the ripper is great at getting full resolutions of older pictures, it seems to have some issues with newer ones.

Ex: https://www.deviantart.com/nivarra/art/COM-Carmelita-CanCan-798781565 https://www.deviantart.com/nivarra/art/COM-Kiss-Pounce-780320385

The full resolution of the first picture is 1200x1200, but the download res is 600x600. While using the ripper on the gallery, the ripped picture is the smaller 600x600.

The second picture is a few months older, with full res of 1500x995, and a download res of 800x531. The ripper getting this picture ripped it at the full res.

It looks like that after March 21, 2019 or so, the site was reconfigured in a way that ripping full res is not possible unless the artist sets it up so. Was just wondering if there was anything that could be done.

wankio commented 5 years ago

I think it can't if download is hidden by artist/uploader

Hrxn commented 5 years ago

So, basically like that "trick" with GIF files? This site is a mess...

mikf commented 5 years ago

I doubt there is a "trick" available this time. Until ID 790677560 it was possible to rewrite the URLs of non-downloadable deviations so they would yield a full resolution image (See 5ec55ec, 9587aea, and https://github.com/r888888888/danbooru/issues/4069), but that doesn't work for newer images and now we are stuck with re-encoded and small JPGs, unless someone finds a way ...

Hrxn commented 5 years ago

The full resolution of the first picture is 1200x1200, but the download res is 600x600. While using the ripper on the gallery, the ripped picture is the smaller 600x600.

But this implies that the 1200x1200 is accessible from the browser? Or not?

biznizz commented 5 years ago

But this implies that the 1200x1200 is accessible from the browser? Or not?

The way it works is that, the download button has two resolutions. One is the smaller resolution that you'd normally get by pressing the button (some times there isn't a button, but I have a browser script that forces the download button to show), and the other is the full resolution of the pic.

So the button would show "600x600 (1200x1200)".

For older pictures, the ripper would get the second, full resolution picture. But because DA changed their site configuration (again) a few months ago, the pictures posted after this would only get ripped at in the smaller resolution.

It was a pleasant surprise to see this ripper was capable of getting (older pictures') full resolution when that wasn't normally possible.

biznizz commented 5 years ago

Also, think I've noticed something. If a picture has a DA watermark on it and it's also shrunken to a smaller size, using the ripper will only scrape the small, watermarked pic. Don't know if this is always how it's been, but it's a strange thing I've noticed.

Hrxn commented 5 years ago

[..] some times there isn't a button, but I have a browser script that forces the download button to show [..]

What kind of browser script is this?

biznizz commented 5 years ago

What kind of browser script is this?

It was something I found on Greasyfork.

https://greasyfork.org/en/scripts/4679-ts-deviantart-download-link

biznizz commented 5 years ago

Well, just noticed another issue. This picture was uploaded a year ago, so it shouldn't have any issues unlike the new uploads this year: https://www.deviantart.com/transflame/art/I-want-to-be-HUMAN-again-765882538

Using gallery on it rips a picture that's 774x7740, but the normal download res is 1024x10240, and the full res is 1445x14550.

Is there a size limit to pics that gallery-dl can rip?

Hrxn commented 5 years ago

Well, the limit is definitely not with gallery-dl, or what do you mean?

biznizz commented 5 years ago

I mean, I'm not sure. I've used the ripper to get other very big pics off DA with no problem; and this is the first time that I've noticed this thing happening, where the ripper downloaded such a super large picture and it's been nearly reduced to half it's original resolution.

I've tried running the process on this pic several times, and it's the greatly reduced picture everytime. I've never touched the py file in the extractor folder, and there shouldn't be any issues with my config file that causes this.

"deviantart": { "refresh-token": "redacted", "flat": true, "folders": false, "journals": "html", "mature": true, "metadata": false, "original": true, "quality": 100, "extra": true, "wait-min": 0

mikf commented 5 years ago

For older DeviantArt images that don't allow an original file download, gallery-dl rewrites their URL to get an "intermediary" version that, 99% of the time, is higher quality than before - except here. I'm guessing this has something to do with the image's aspect ratio and a height limit of the "intermediary" version, maybe?

To give a concrete example: The image URL presented by DeviantArt is

https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/f7f803e5-c49e-4d9c-ba43-a8e7df90a07e/dcnziqy-dab1058a-fca5-4511-bd73-002789d58ccd.png/v1/fill/w_1024,h_10240,strp/i_want_to_be_human_again_by_transflame_dcnziqy-fullview.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7ImhlaWdodCI6Ijw9MTAyNDAiLCJwYXRoIjoiXC9mXC9mN2Y4MDNlNS1jNDllLTRkOWMtYmE0My1hOGU3ZGY5MGEwN2VcL2RjbnppcXktZGFiMTA1OGEtZmNhNS00NTExLWJkNzMtMDAyNzg5ZDU4Y2NkLnBuZyIsIndpZHRoIjoiPD0xMDI0In1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmltYWdlLm9wZXJhdGlvbnMiXX0.0a7GgC31lMDQmKusL9bsvz2JdjlpIxRs1N7J-STX1Mk

and gallery-dl transforms it to the following, hoping to get a better version of that image:

https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/intermediary/f/f7f803e5-c49e-4d9c-ba43-a8e7df90a07e/dcnziqy-dab1058a-fca5-4511-bd73-002789d58ccd.png

Is there a way to search by image resolution / aspect ratio on DeviantArt so I can figure out where the "cutoff" point is?

ShapeShifter499 commented 5 years ago

I'm confused by "quality": 100,

The man page suggests that setting means 100% compression which I think is a bad thing.

Does that setting mean 100 percent quality (as close as original as you can get) or 100 percent compression? I'd rather be downloading as close to original as possible.

Hrxn commented 5 years ago

What do you mean? This?

JPEG compression quality for newer images hosted on wixmp servers.

Yea, this is a quality setting, not compression level or something, so q=100 is the highest you can use for dA still returning an image. This is all server-side, though, i.e. it is still somewhat compressed, obviously.

ShapeShifter499 commented 5 years ago

@Hrxn I'm trying to download images I uploaded to my own account and I wanted to ensure I got the highest quality as I already sent. I'm only doing this because my computer had a hardware failure before I could backup and I wanted to make backups.

I have enabled oauth login

Hrxn commented 5 years ago

Ah, that is a good question. I am not sure if the original/raw files are publicly accessible via the site itself. But somewhere in the profile settings should be an option to export the complete profile data. I mean it's your data, after all. And if you still find a local file, you could compare checksums against a file downloaded from dA.

ShapeShifter499 commented 5 years ago

@Hrxn I just said I had a hardware failure. If I could compare against a local file I would but I can't.

Hrxn commented 5 years ago

I'm aware.. but you maybe have some files saved to another data storage device (flash drive, disc, whatever), or send some files directly to a friend or something.

ShapeShifter499 commented 5 years ago

@Hrxn only a small set of files. Most were one time downloads I managed to upload but lost the local copy for. Commissions or freebie art I got. I only was doing this for a couple of months. I got lazy on backup and organize. Then I went and spilled coffee all over my laptop killing the one location these files were on.

mikf commented 5 years ago

@ShapeShifter499 If you haven't disabled the Free Download option when uploading your images to DeviantArt, gallery-dl will download the original, unaltered files. quality doesn't apply in those cases. It only gets used for images where the original download has been disabled. (See #369)

kattjevfel commented 3 years ago

For what it's worth the second link (https://www.deviantart.com/nivarra/art/COM-Kiss-Pounce-780320385) now downloads the proper 1500x995 file, but the first link still downloads a 600x600 file.

mikf commented 3 years ago

As it turns out, there is a way to get the full resolution version for non-downloadable images like https://www.deviantart.com/nivarra/art/COM-Carmelita-CanCan-798781565:

https://github.com/mikf/gallery-dl/commit/02a247f4e54b6835e81102b84583bdae0969a050

Someone clever, i.e. not me, figured out a way to produce a valid token while also removing the width and height limits for dA image URLs.

No Idea how long it takes to get "fixed", so grab it while it's hot, I guess. See https://github.com/mikf/gallery-dl/commit/02a247f4e54b6835e81102b84583bdae0969a050#commitcomment-58578639 for how to update your collection.

biznizz commented 2 years ago

I don't know any computer language stuff and tried everything but can't understand how to do it. Can someone please download a single old DeviantArt picture (2012) for me in original resolution(1800x2466px 4.79 MB) and mail me that ?

Here. I put it on Imgur for you, I'm not doing an email. https://imgur.com/a/AwZW4Nz

Hrxn commented 2 years ago

Is there anything unusual about this specific image?

biznizz commented 2 years ago

Is there anything unusual about this specific image?

No, it downloaded just fine, at full res, even with the author selling prints of it for a few bucks. I think dude just didn't want to deal with the hassle of trying to install the program, so I just nipped this thing in the bud by giving him the pic he wanted.

mikf commented 1 year ago

The method implemented in https://github.com/mikf/gallery-dl/commit/02a247f4e54b6835e81102b84583bdae0969a050 seems to no longer work:

$ gallery-dl https://www.deviantart.com/littlewickedowl
[downloader.http][warning] '401 Unauthorized' for 'https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/0a3b1c1b-05a6-4bdd-bf4a-82977cbdd2ba/dg8f0jd-51bb0861-b4e6-4a48-b7ea-e27d90356ab2.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mLzBhM2IxYzFiLTA1YTYtNGJkZC1iZjRhLTgyOTc3Y2JkZDJiYS9kZzhmMGpkLTUxYmIwODYxLWI0ZTYtNGE0OC1iN2VhLWUyN2Q5MDM1NmFiMi5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ.'
[download][info] Trying fallback URL #1
/tmp/_/deviantart/LittleWickedOwl/devia…t_981596281_No. 563  - Another Realm.jpg
/tmp/_/deviantart/LittleWickedOwl/deviantart_981402996_Lilac Bush In Winter.jpg
[downloader.http][warning] '401 Unauthorized' for 'https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/0a3b1c1b-05a6-4bdd-bf4a-82977cbdd2ba/dg8ao7q-86269d3c-2dbb-4997-b638-41be3a8b62e7.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mLzBhM2IxYzFiLTA1YTYtNGJkZC1iZjRhLTgyOTc3Y2JkZDJiYS9kZzhhbzdxLTg2MjY5ZDNjLTJkYmItNDk5Ny1iNjM4LTQxYmUzYThiNjJlNy5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ.'
[download][info] Trying fallback URL #1
/tmp/_/deviantart/LittleWickedOwl/deviantart_981393686_No. 558.jpg
Twi-Hard commented 1 year ago

Is there any hope in fixing it? I tested a non-downloadable image and watermarked image and both didn't work. Downloading this paid image worked though https://www.deviantart.com/darkflame75/art/The-Moon-Rises-603249824