dataabc / weiboSpider

新浪微博爬虫,用python爬取新浪微博数据
8.37k stars 1.98k forks source link

printer.print_user_info函数中偶尔报错停下程序 #167

Closed PhoebusSi closed 4 years ago

PhoebusSi commented 4 years ago

为了更好的解决问题,请认真回答下面的问题。等到问题解决,请及时关闭本issue。

dataabc commented 4 years ago

感谢反馈。

应该是被限制了,限制一段时间会自动解除,可以通过降低速度解决,具体修改weibo_spider.py文件的get_weibo_info方法,原来的代码:

                    if (page - page1) % random_pages == 0 and page < page_num:
                        sleep(random.randint(6, 10))
                        page1 = page
                        random_pages = random.randint(1, 5)

上面是每1到5页随机等待6-10秒,你可以加快暂停频率(减小random_pages)或加大等待时间(加大sleep)解决。

如果还有问题,欢迎继续讨论

PhoebusSi commented 4 years ago

谢谢 我在另一个issue中找到了这个原因。感谢感谢

信息科学学院佀庆一

邮箱:xk17sqy@126.com |

签名由 网易邮箱大师 定制

在2020年06月13日 19:38,Chen Lei 写道:

感谢反馈。

应该是被限制了,限制一段时间会自动解除,可以通过降低速度解决,具体修改weibo_spider.py文件的get_weibo_info方法,原来的代码:

                if (page - page1) % random_pages == 0 and page < page_num:
                    sleep(random.randint(6, 10))
                    page1 = page
                    random_pages = random.randint(1, 5)

上面是每1到5页随机等待6-10秒,你可以加快暂停频率(减小random_pages)或加大等待时间(加大sleep)解决。

如果还有问题,欢迎继续讨论

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 4 years ago

Closing as stale, please reopen if you'd like to work on this further.