Closed damare01 closed 2 years ago
lightnovelpub seems to block consecutive requests for an IP after downloading 200 or so chapters.
to solve this issue we can consider using some free proxy servers
I found a nice package: https://pypi.org/project/free-proxy/
Unfortunately it has become so popular that https proxies are hard to acquire. But there are lots of http proxies available.
Now the crawler will automatically change the proxy servers every 30s interval.
Let us know
Novel URL: https://www.lightnovelpub.com/novel/bank-of-the-universe-12032016 App Location: Discord and Telegram App Version: 2.29.7
Describe this issue
Unable to crawl