readwn.com: HTTPError: 429 Client Error: Too Many Requests for url

[stbooks]

Let us know

Novel URL: [ ]https://www.readwn.com/novel/peerless-martial-soul.html App Location: PIP | EXE | Discord | Telegram EXE App Version: v3.2.2

Describe this issue

It will not crawl anymore after several pages and state too many requests. Please fix it. Thks

[zerty]

@dipu-bd Hey, would using the ratelimit module be possible? I've tried it and it works more or less (still need fine tuning) => 02ded725a361c7a4c9563891b16da0858a1407e8

If you accept this kind of solution I'll make a pull request after some edits

[dipu-bd]

I like the solution with decorators. However it would be better to put in in the Crawler class. Let me take over this task to do the necessary modifications

[NightMachinery]

This problem also exists for novelbin:

lightnovel-crawler --all --single --format mobi --suppress --source 'https://novelbin.com/b/lord-of-mysteries-2-circle-of-inevitability#tab-chapters-title'

v3.2.6
╱╱╱╱╱╰━━╯ 🔗 https://github.com/dipu-bd/lightnovel-crawler
--------------------------------------------------------------------------------
 ❗ Input is suppressed
--------------------------------------------------------------------------------
Namespace(log=None, log_file=None, list_sources=False, crawler=[], novel_page='https://novelbin.com/b/lord-of-mysteries-2-circle-of-inevitability#tab-chapters-title', query=None, login=None, output_formats=['mobi'], add_source_url=False, single=True, multi=False, output_path=None, filename=None, filename_only=False, force=False, ignore=False, all=True, first=None, last=None, page=None, range=None, volumes=None, chapters=None, proxy_file=None, auto_proxy=False, bot=None, shard_id=0, shard_count=1, selenium_grid=None, suppress=True, close_directly=False, extra={})

➡ Press  Ctrl + C  to exit

Retrieving novel info...

📒 Lord Of Mysteries 2: Circle Of Inevitability
3 volumes and 252 chapters found.
🔗 https://novelbin.com/b/lord-of-mysteries-2-circle-of-inevitability#tab-chapters-title

HTTPError: 429 Client Error: Too Many Requests for url: https://novelusb.com/novel-book/lord-of-mysteries-2-circle-of-inevitability-novel/cchapter-20-chapter-20-customs
HTTPError: 429 Client Error: Too Many Requests for url: https://novelusb.com/novel-book/lord-of-mysteries-2-circle-of-inevitability-novel/chapter-25-sequences-and-potions
HTTPError: 429 Client Error: Too Many Requests for url: https://novelusb.com/novel-book/lord-of-mysteries-2-circle-of-inevitability-novel/chapter-26-whistleblowing
HTTPError: 429 Client Error: Too Many Requests for url: https://novelusb.com/novel-book/lord-of-mysteries-2-circle-of-inevitability-novel/chapter-27-five-changes
HTTPError: 429 Client Error: Too Many Requests for url: https://novelusb.com/novel-book/lord-of-mysteries-2-circle-of-inevitability-novel/chapter-28-laws
...
HTTPError: 429 Client Error: Too Many Requests for url: https://novelusb.com/novel-book/lord-of-mysteries-2-circle-of-inevitability-novel/cchapter-250-250-condolence-banquet
Chapters: 100%|██████████████████████████████████| 252/252 [01:28<00:00,  2.86item/s]
  Images: 100%|████████████████████████████████████| 1/1 [00:00<00:00, 5817.34item/s]
Created: Lord Of Mysteries 2 Circle Of Inevitability c1-252.epub
[mobi] conversion failed: Lord Of Mysteries 2 Circle Of Inevitability c1-252.epub
✨ Task completed