search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.82k stars 970 forks source link

Unable to download all images from deviantart and pixiv search results #1704

Open Coca836 opened 3 years ago

Coca836 commented 3 years ago

Hi, I'd like to thank you for your help.deviantartpixivI have downloaded from the search results on the above sites.However, the number of downloaded files is significantly different from the search results.On deviantart, I have more than 3k search results, but only around 2000 are downloaded.On pixiv, the difference is around 10000. Is there anything I can do to fix this?

mikf commented 3 years ago

No, there isn't. Pixiv search is limited to the first 5k results. The DeviantArt code uses a different API endpoint than the website itself, which doesn't return the same results and might also be limited.

Coca836 commented 3 years ago

I'm sorry to hear that. I'd like to download all the search results, but if it's a specification, there's nothing I can do about it. Is it also impossible to download all of the images for a target keyword? I'm hoping that the ability to download all images will be added.

nisehime commented 3 years ago

Can't you use the web api for search though?

mikf commented 3 years ago

For Pixiv? No, the results from the web API are vastly different than from the mobile API, it is noticeably slower, and it would require a different kind of authentication (cookies instead of OAuth tokens)

For DeviantArt? Maybe, but it would have the same problems as the current way of fetching scraps: Extra API calls, inability to fetch artworks rated "mature" without being logged in (through cookies), and the possibility of getting logged out and missing "mature" artworks halfway through a run.

I do want to transition to dA's web API at some point, but that is going to cause all sort of problems and bugs when that happens.

Coca836 commented 3 years ago

It's a very difficult problem. When I download with DeviantArt, it stops downloading after less than 100 files, even if I have more than a thousand files. Is it possible to set a delay for the download?

By the way, has the specification of gallery-dl been changed? When I download from Twitter search results, it creates a lot of folders. It used to just create a folder with the keyword name under gallery-dl.

mikf commented 3 years ago

Is it possible to set a delay for the download?

With the sleep/--sleep or sleep-request options, but I don't think that's really going to help. If you are getting repeated 429 errors, try getting your own API credentials.

By the way, has the specification of gallery-dl been changed? When I download from Twitter search results, it creates a lot of folders. It used to just create a folder with the keyword name under gallery-dl.

Twitter search now uses the same directory format as all other Twitter extractors (a751afdfb37eb2b1ce376eabe1ed3fe15c620b71). You can change it back if the old method was more useful for you: https://github.com/mikf/gallery-dl/issues/1308#issuecomment-866124683

Coca836 commented 3 years ago

Thanks for the reply. I tried that. However, the delay setting doesn't seem to work.

https://github.com/mikf/gallery-dl/issues/1308#issuecomment-866124683 I read it. How can I do this? I have looked into it, but I don't understand it. I am a beginner.