avluis / Hentoid

Doujinshi Android App
https://discord.gg/QEZ3qk9
Apache License 2.0
1.08k stars 85 forks source link

E-hentai : Better handle large galleries #404

Closed RobbWatershed closed 3 years ago

RobbWatershed commented 5 years ago

If you try queuing huge books (800+ pages), what you describe might just happen The app "scouts" all the pages to prepare the download However, when there are too many of them, it encounters e-hentai's download limit and fails

User Azoz5643 report preparation phase stopping by itself.

RobbWatershed commented 4 years ago

e-hentai images have a very specific URL that cannot be guessed at all - you need to open each page of the gallery to know it that's what happens during preparation phase : the app opens every page of the gallery to pre-fetch image URLs and feed the download engine with them as you guessed, I believe -not certain but 80% sure- that opening these pages counts against the daily download limit that's why we have this issue open the idea behind this issue is to be more "subtle" with that whole preparation phase for instance, fetch the first 20 pages, then download them, then fetch the 20 next and download them...

the big unknown on e-hentai is what happens server-side their content is not hosted centrally but by the e-h community, on hundreds of servers around the world as a consequence, a valid URL on day N might be unavailable on day N+1 I believe that means we always have to "scout" album pages before downloading them, even if they have been previously discovered but I'm not totally sure about that either - I need to investigate further

RobbWatershed commented 4 years ago

Doing this would require redesigning / rewriting the download engine, possible around RxJava, so that the preparation phase would disappear and be replaced by a "feeding" mechanism for the main download loop.

This is too big to be developed solely for eHentai / exHentai and should be coupled with another need that also requires such a redesign.

RobbWatershed commented 3 years ago

Coupled with #777 for v1.17

RobbWatershed commented 3 years ago

Done during #277