search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.34k stars 923 forks source link

Twitter - Randomly Downloading Old Media #4822

Open Soeroah opened 10 months ago

Soeroah commented 10 months ago

For the last several days, each day I run my batch file to download media from a bunch of art accounts (check for already done, stop at the first hit, otherwise download the most recent 30) it seems to pick a random artist and just downloads 30 images from that account from years ago, well over 50 media posts ago. Essentially I'm used to it only checking any images that were not downloaded since the last time I ran the batch and stopping once it hits one that downloaded previously. I'm used to 0-3 images each time, but now it picks a seemingly random account and downloads as much as 30.

I'm not really sure why it's suddenly doing this. The prompts have worked flawlessly for months.

mikf commented 10 months ago

Probably Twitter having some (internal) issues, again. gallery-dl only tries to download from whatever Tweets Twitter's API returns. It handles each user the same and doesn't do anything special at random, unless the data it gets is "random".

Soeroah commented 10 months ago

That's a disappointing answer for the purposes of my productivity but it's an extremely believable one, too. Hopefully it either gets back to working soon or bluesky miraculously re-designs itself to be gallery-DL-able

Thanks

Soeroah commented 10 months ago

Really sorry to re-open this already, but I just did my pull for the day and it happened for a deviant art account this time, so it doesn't seem to be a twitter thing... is it possible my archive file is purging old information or something? It's weird that it only seems to happen once a day

mikf commented 9 months ago

I can't think of any way this would happen for dA, unless you somehow used --range. The archive file "losing" entries is also not a thing that could happen with gallery-dl alone. There is no code to remove anything from an archive, only check and add.

Could you post your config file here with user/pass/etc removed?

Soeroah commented 9 months ago

Here's the config as requested. The last time it was edited was sometime in May, so I'm not sure why it'd suddenly start having issues in November, but I do use --range for my batches, cuz I want each account to grab the last 30 but stop if any duplicates are met - which usually means only a couple hits a day.

config.json

The only sites I currently use are Pixiv, Twitter and DeviantArt