shengjiangfeng
commented
3 months ago 微信读书会反爬， 部分章节没有下一章和下一页显示，导致爬虫load超时，暂时未找到微信读书是怎么识别这个爬虫的
Open mafeiduoduo opened 3 months ago
shengjiangfeng
commented
3 months ago 微信读书会反爬， 部分章节没有下一章和下一页显示，导致爬虫load超时，暂时未找到微信读书是怎么识别这个爬虫的
我在使用该项目时部分能正常运行，但是遇到内容较多的书籍时会出现（RuntimeError: Wait for creating markdown timeout）。不是太敢多次尝试，想知道是什么情况？是否可以继续使用？（害怕封号）。是否有其他解决方法？ [2024-06-17 16:15:42,607][INFO][WeReadWebPage] Fetch url https://weread-1258476243.file.myqcloud.com/web/wrwebnjlogic/js/app.aecf9799.js Traceback (most recent call last): File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/duoduo/下载/weread-exporter-main/weread_exporter/main.py", line 158, in
main()
File "/home/duoduo/下载/weread-exporter-main/weread_exporter/main.py", line 154, in main
loop.run_until_complete(async_main())
File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/home/duoduo/下载/weread-exporter-main/weread_exporter/main.py", line 92, in async_main
await exporter.export_markdown(args.load_timeout, args.load_interval)
File "/home/duoduo/下载/weread-exporter-main/weread_exporter/export.py", line 353, in export_markdown
markdown = await self._page.get_markdown()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/duoduo/下载/weread-exporter-main/weread_exporter/webpage.py", line 400, in get_markdown
raise RuntimeError("Wait for creating markdown timeout")
RuntimeError: Wait for creating markdown timeout
[2024-06-17 16:15:56,221][INFO]terminate chrome process...