SpiderClub / weibospider

:zap: A distributed crawler for weibo, building with celery and requests.
MIT License
4.81k stars 1.21k forks source link

启动worker时执行到**[2020-04-02 12:36:58,850: INFO/MainProcess] mingle: all alone**就不再继续 #204

Open keithkang1986 opened 4 years ago

keithkang1986 commented 4 years ago

在提交Issue之前请先回答下面问题,谢谢!

1.你是怎么操作的?

按步骤到第9步 然后执行celery -A tasks.workers -Q login_queue,user_crawler,fans_followers,search_crawler,home_crawler worker -l info -c 1

2.你期望的结果是什么? 启动worker

3.实际上你得到的结果是什么? C:\data\爬虫\weibospider-1.7.2\config\conf.py:12: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://m sg.pyyaml.org/load for full details. cf = load(cont)

-------------- celery@LAPTOP-QD49LR5G v4.4.2 (cliffs) --- * ----- -- *** ---- Windows-10-10.0.18362-SP0 2020-04-02 12:36:57

[tasks] . tasks.comment.crawl_comment_by_page . tasks.comment.crawl_comment_page . tasks.comment.excute_comment_task . tasks.home.crawl_ajax_page . tasks.home.crawl_weibo_datas . tasks.home.excute_home_task . tasks.login.excute_login_task . tasks.login.login_task . tasks.repost.crawl_repost_by_page . tasks.repost.crawl_repost_page . tasks.repost.excute_repost_task . tasks.search.excute_search_task . tasks.search.search_keyword . tasks.user.crawl_follower_fans . tasks.user.crawl_person_infos . tasks.user.excute_user_task

[2020-04-02 12:36:57,822: INFO/MainProcess] Connected to redis://127.0.0.1:6379/5 [2020-04-02 12:36:57,830: INFO/MainProcess] mingle: searching for neighbors [2020-04-02 12:36:58,236: INFO/SpawnPoolWorker-1] child process 13708 calling self.run() C:\data\爬虫\weibospider-1.7.2\config\conf.py:12: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://m sg.pyyaml.org/load for full details. cf = load(cont) [2020-04-02 12:36:58,850: INFO/MainProcess] mingle: all alone

4.你使用的是哪个版本的WeiboSpider? 你的操作系统是什么?是否有读本项目的[常见问题] weibospiderv1.7.2;操作系统windows10。已读相关问题,但未解决。

thekingofcity commented 4 years ago

windows并非项目支持, 尝试降级下celery? 我记得4.x就抛弃windows了

OneCodeMonkey commented 4 years ago

@keithkang1986 win10 下可以起 虚拟机开发,直接拿windows 的 python 环境跑不行的