mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
10.87k stars 888 forks source link

Deviantart: Is it possible to avoid these API calls when skipping downloads? #4767

Open tcmbackwards opened 8 months ago

tcmbackwards commented 8 months ago

I'm hitting the API limit quickly despite downloads being skipped because gallery-dl seems to still be making API calls for every skipped download. It doesn't happen with every gallery but here's an example. Is it possible to avoid these calls?


[gallery-dl][debug] Version 1.26.1 - Executable
[gallery-dl][debug] Python 3.8.10 - Windows-10-10.0.19045
[gallery-dl][debug] requests 2.31.0 - urllib3 1.26.17
[gallery-dl][debug] Configuration Files ['G:\\gallery-dl\\gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/vermillianno/gallery'
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/vermillianno/gallery'
[deviantart][debug] Using custom API credentials (client-id ****)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/vermillianno HTTP/1.1" 200 1328
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=vermillianno&offset=0&limit=24&mature_content=true HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/metadata?deviationids%5B0%5D=D38DB198-CEFD-BBA9-B721-EFFD742B24D4&deviationids%5B1%5D=9E965F12-5DE0-8EB1-5E3C-69C9D49080EB&deviationids%5B2%5D=B209F0C6-9F23-B9E8-ED21-DC84B61A48FA&deviationids%5B3%5D=CF3424FE-0162-7FE2-FD1E-ED506799DA28&deviationids%5B4%5D=D53ACFA5-03E4-CB0A-326A-1B4F61A8D3F2&deviationids%5B5%5D=553072EA-18EF-569B-0D9C-F465F87C4438&deviationids%5B6%5D=011E3F22-4852-D240-BEC5-357373C09095&deviationids%5B7%5D=AA93E7A4-07E6-5C90-B242-5000C6B998CF&deviationids%5B8%5D=F02D6D0A-908C-9EE2-AC6F-C861784DFCBA&deviationids%5B9%5D=3E6998F4-DD20-367B-3727-A839CFE35EA4&deviationids%5B10%5D=0D6F6714-191D-CB40-8C45-66BD519E5AE9&deviationids%5B11%5D=28CD06A4-2863-BCA1-D4B4-CB466C09D926&deviationids%5B12%5D=6D15E6FA-6C80-82CC-49B6-04657529C544&deviationids%5B13%5D=3F552B86-F587-754E-922A-B494B3D8C653&deviationids%5B14%5D=82F4F5B1-E8D6-66C2-B06E-C66E165AE31B&deviationids%5B15%5D=98EFF86A-91B8-70D0-7DC8-ED29A90C8B5E&deviationids%5B16%5D=553124DB-CC83-5CA6-BA63-7D36C96940EC&deviationids%5B17%5D=5721E7E3-C3FF-A10B-BC35-7EC0489EF12F&deviationids%5B18%5D=D5454D4A-7BD6-8744-798C-3FB674CADC2E&deviationids%5B19%5D=6F623E23-9237-07CA-8EF3-58A1A6FAA3A8&deviationids%5B20%5D=73043376-C1D0-A992-F2CC-F4792EA05FF4&deviationids%5B21%5D=30159B45-5915-73BE-429D-D925DF11FD3B&deviationids%5B22%5D=E11EE3B3-4D78-C4E0-9FB4-283A26D3E959&deviationids%5B23%5D=E39E2695-BD51-01E2-34C5-59F0CFB7026E&mature_content=true HTTP/1.1" 200 1982
[deviantart][debug] Using download archive 'deviantart-vermillianno-archive.sqlite3'
[postprocessor.metadata][debug] Using metadata archive 'deviantart-vermillianno-metaarchive.sqlite3'
[deviantart][debug] Active postprocessor modules: [MetadataPP]
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/D38DB198-CEFD-BBA9-B721-EFFD742B24D4?mature_content=true HTTP/1.1" 200 594
# .\gallery-dl\deviantart\vermillianno\deviantart_992202208_old man.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/9E965F12-5DE0-8EB1-5E3C-69C9D49080EB?mature_content=true HTTP/1.1" 200 592
# .\gallery-dl\deviantart\vermillianno\deviantart_991901359_Howl.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/B209F0C6-9F23-B9E8-ED21-DC84B61A48FA?mature_content=true HTTP/1.1" 200 596
# .\gallery-dl\deviantart\vermillianno\deviantart_984270208_ghosts.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/CF3424FE-0162-7FE2-FD1E-ED506799DA28?mature_content=true HTTP/1.1" 200 597
# .\gallery-dl\deviantart\vermillianno\deviantart_973726104_More Athan.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/D53ACFA5-03E4-CB0A-326A-1B4F61A8D3F2?mature_content=true HTTP/1.1" 200 593
# .\gallery-dl\deviantart\vermillianno\deviantart_973725937_Athan.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/553072EA-18EF-569B-0D9C-F465F87C4438?mature_content=true HTTP/1.1" 200 614
# .\gallery-dl\deviantart\vermillianno\deviantart_973725446_yet another drawing of lucky.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/011E3F22-4852-D240-BEC5-357373C09095?mature_content=true HTTP/1.1" 200 599
# .\gallery-dl\deviantart\vermillianno\deviantart_973725184_Miles Morales.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/AA93E7A4-07E6-5C90-B242-5000C6B998CF?mature_content=true HTTP/1.1" 200 601
# .\gallery-dl\deviantart\vermillianno\deviantart_973725060_A building.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/F02D6D0A-908C-9EE2-AC6F-C861784DFCBA?mature_content=true HTTP/1.1" 200 610
# .\gallery-dl\deviantart\vermillianno\deviantart_973724817_Chroma corps cadet_ Calico.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/3E6998F4-DD20-367B-3727-A839CFE35EA4?mature_content=true HTTP/1.1" 200 595```

Here's my config file, I'm making a metadata archive but I can't see that it's doing anything. Setting "extra" to false didn't help.

```{
    "extractor": {
        "archive": "{category}-{username}-archive.sqlite3",
        "archive-pragma": ["journal_mode=WAL", "synchronous=NORMAL"],

        "deviantart": {
            "client-id": "****",
            "client-secret": "****",
            "auto-watch": true,
            "extra": true,
            "include": "gallery,scraps,journals",
            "public": false,
        "sleep": 2,
            "postprocessors": [{
                    "name": "metadata",
            "skip": true,
            "archive": "{category}-{username}-metaarchive.sqlite3",
            "archive-pragma": ["journal_mode=WAL", "synchronous=NORMAL"]
                }
            ]
        }
    },
    "downloader": {},
    "output": {},
    "postprocessor": {}
}```
mikf commented 8 months ago

You can use --range 123- to skip that many posts ahead without using any extra API calls. Avoiding API calls for skipped downloads is currently not possible.

mikf commented 7 months ago

Duplicate of #317