search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

Deviantart: Sometimes incomplete galleries #346

Closed petrarislin closed 5 years ago

petrarislin commented 5 years ago

1) NSFW. Occasionally, under win10 gallery-dl will appear to complete a gallery without a problem, but hasn't downloaded all the photos. Example on https://www.deviantart.com/bondagedaydreams/gallery/7836052/hotel-room-bondage downloaded only 18 of the 24 rather similar images on the first page, and did not process the second page: C:\gallery-dl>gallery-dl -v https://www.deviantart.com/bondagedaydreams/gallery/7836052/hotel-room-bondage [gallery-dl][debug] Version 1.9.0 [gallery-dl][debug] Python 3.7.3 - Windows-10-10.0.17763 [gallery-dl][debug] requests 2.22.0 - urllib3 1.24.3 [gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/bondagedaydreams/gallery/7836052/hotel-room-bondage' [gallery-dl][debug] Updating urllib3 ciphers [deviantart][debug] Using DeviantartFolderExtractor for 'https://www.deviantart.com/bondagedaydreams/gallery/7836052/hotel-room-bondage' [urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443 [urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/bondagedaydreams HTTP/1.1" 200 1047 [urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/folders?username=bondagedaydreams&offset=0&limit=50&mature_content=true HTTP/1.1" 200 1460 [urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/C4675812-23D2-8094-14FD-3F3533BAE23B?username=bondagedaydreams&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 5933 .\gallery-dl\deviantart\bondagedayd...tart_105510830_Hotel Room Bondage I.jpg .\gallery-dl\deviantart\bondagedayd...art_105511143_Hotel Room Bondage II.jpg .\gallery-dl\deviantart\bondagedayd...rt_105511459_Hotel Room Bondage III.jpg .\gallery-dl\deviantart\bondagedayd...art_105511835_Hotel Room Bondage IV.jpg .\gallery-dl\deviantart\bondagedayd...tart_105771687_Hotel Room Bondage V.jpg .\gallery-dl\deviantart\bondagedayd...art_105772200_Hotel Room Bondage VI.jpg .\gallery-dl\deviantart\bondagedayd...rt_105822175_Hotel Room Bondage VII.jpg .\gallery-dl\deviantart\bondagedayd...t_105822669_Hotel Room Bondage VIII.jpg .\gallery-dl\deviantart\bondagedayd...art_106672389_Hotel Room Bondage IX.jpg .\gallery-dl\deviantart\bondagedayd...tart_106672559_Hotel Room Bondage X.jpg .\gallery-dl\deviantart\bondagedayd...art_106672909_Hotel Room Bondage XI.jpg .\gallery-dl\deviantart\bondagedayd...rt_106673346_Hotel Room Bondage XII.jpg .\gallery-dl\deviantart\bondagedayd...t_106673709_Hotel Room Bondage XIII.jpg .\gallery-dl\deviantart\bondagedayd...rt_107057423_Hotel Room Bondage XIV.jpg .\gallery-dl\deviantart\bondagedayd...art_107057813_Hotel Room Bondage XV.jpg .\gallery-dl\deviantart\bondagedayd...rt_117583103_Hotel Room Bondage XVI.jpg .\gallery-dl\deviantart\bondagedayd...t_117587378_Hotel Room Bondage XVII.jpg .\gallery-dl\deviantart\bondagedayd..._117587992_Hotel Room Bondage XVIII.jpg [urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/C4675812-23D2-8094-14FD-3F3533BAE23B?username=bondagedaydreams&offset=24&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 70 (edited out the hashtags at the beginning of each jpg line since it interfered with the formatting of this post)

2) Occasionally, setting it to download a deviantart page downloads the sub-galleries also, each in their own folder, while usually it just downloads them all into the main folder. Is there a way to force the full folder tree download?

TY.

mikf commented 5 years ago

I somehow managed to miss your issue, sorry about that.

  1. Some deviations are private and can only accessed when using an OAuth token. You can get one with gallery-dl oauth:deviantart

  2. Occasionally, setting it to download a deviantart page downloads the sub-galleries also, each in their own folder, while usually it just downloads them all into the main folder.

    This shouldn't or actually can't happen. Do you have some example URLs where it does and doesn't do this?

    Is there a way to force the full folder tree download?

    Set extractor.deviantart.flat to false, but be aware that this might miss deviations or download them multiple times depending on how the gallery's folder structure was set up.