search

KurtBestor / Hitomi-Downloader

:cake: Desktop utility to download images/videos/music/text from various websites, and more.
22.4k stars 2.06k forks source link

Pixiv grabbing speed is too slow #3232

Open nisehime opened 3 years ago

nisehime commented 3 years ago

Not sure if you saw it, but here: https://github.com/KurtBestor/Hitomi-Downloader/issues/3083#issuecomment-778813421

In the end, it took me 8 hours to finish updating all tasks. The possible solution I see is implementing OAuth authorization, like it was done in gallery-dl.

KurtBestor commented 3 years ago

Well, 281 tasks are quite a lot. I optimized some code, now it's about x4 faster. Try this: https://github.com/KurtBestor/Hitomi-Downloader/releases/tag/Technical-Preview

nisehime commented 3 years ago

Well, 281 tasks are quite a lot.

When there was login system, restarting this amount would take like 30-40 minutes.

Restarted 74 tasks now - it took me 30 minutes. There were many old ugoiras found, which Hitomi didn't download back then for some reasons, so from these 30 minutes you probably can subtract some time wasted on converting them. Anyway, first impression: better, but still not as good as it used to be.

UPD: wait, it didn't find missed ugoiras, it just redownloaded all them again, so it's a bug actually.

KurtBestor commented 3 years ago

It's not a bug; Bug is an error in a system that causes it to produce an unexpected result. I just didn't implement it yet. I'll implement these "features" later.

If there is a feature you want, You can contribute to this project directly: https://github.com/KurtBestor/Hitomi-Downloader/blob/master/src/extractor/pixiv_downloader.py

nisehime commented 3 years ago

It's not a bug; Bug is an error in a system that causes it to produce an unexpected result. I just didn't implement it yet. I'll implement these "features" later.

What do you mean? Every time I restart a pixiv entry which has ugoiras (animated pics), it redownloads all of them again and goes through converting (to gif) process, instead of skipping them since they're already present in the artist's folder. I checked the "date created" in the file properties and it's a new date, so hitomi has indeed overwritten existed file. It never happened before this preview I downloaded yesterday, and it's surely a bug.