manga-download / hakuneko

Manga & Anime Downloader for Linux, Windows & MacOS
https://hakuneko.download
The Unlicense
4.66k stars 466 forks source link

[Webtoons.com] Connector not fetching all chapters for long running titles #6600

Closed Elliander closed 9 months ago

Elliander commented 9 months ago

Is there an existing issue for this connector?

Did you read the troubleshooting guide ?

Is the website of the connector working properly ?

Make sure to try Nightly Build

What kind of issue are you encountering

Other

Describe the bug

Over the past few months I noticed that when a title on "Line Webtoons" has a few hundred chapters that they don't all appear in the download list. Usually it stops after around 100 chapters which would represent 10 pages of the chapter list. This is a problem because I've been trying to download titles before they become "Daily Pass" titles that get locked from viewing on any PC. Initially I tried to work around this issue by selecting a URL for a different page number, but turns out that this didn't really do anything other than refresh the crawler. Pressing the green refresh button and then clicking the title on the list again gives it a chance of fetching more titles which will occasionally fetch the entire list. The more chapters there are the less likely it is that I can fetch them all.

This issue is one that I have not personally observed with other sites though which is why I am reporting this as a connector issue. My guess is that the crawler might just give up after it goes through so many pages which would explain why trying from, say, page 27 (for a title with 270 chapters) would at least have a chance of crawling through everything while selecting the main page only seems to be able to crawl through 100 chapters for downloading. It's still a bit random though.

Note that I paste from URL since "This website does not provide a manga list". Also note that the little spinning circle stops spinning before showing the chapter list so sitting there and waiting wouldn't make a difference.

To Reproduce

  1. Name of connector: Line Webtoons (English)
  2. Name of manga (if connector update is working)

In this last case I was trying to download "Weak Hero"

  1. Name of chapter (if manga update is working)

It's random, but here's the URL of the last title affected by this issue:

https://www.webtoons.com/en/action/weakhero/list?title_no=1726

But even while downloading all the chapters here's a screenshot of all I see on the last try where you can clearly see that in this case the list stops at 49, but my download list goes beyond that. That's better than usual as it tends to stop around 90:

incomplete_chapter_list

  1. Copy any relevant error message that appears in a popup or (advanced) within the console (press F12)

No error messages appear, although, and this may be unrelated, while repeatedly trying to get all chapters my bookmarks were all deleted.

MikeZeDev commented 9 months ago

The problem with LineWebtoon is that their chapter list is paginated, meaning more requests to get all of them . I see you downloaded quite a lot of chapters, you must have hit the rate limit set up on their server. This should also explain the randomness of this problem.

There is nothing we can do. Make sure to use nightly and setup a throttling delay in settings for this website

image