search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 323 forks source link

cluster mode online test hangs #213

Closed danmsf closed 5 years ago

danmsf commented 5 years ago

HI Im trying the DIY cluster mode, and Im running into some issues running the online test. I set info to debug and this is where it hangs without throwing an error or response:

2019-01-22 10:51:40,520 [kafka-monitor] DEBUG: Successfully connected to Kafka
2019-01-22 10:51:40,520 [kafka-monitor] DEBUG: Trying to load plugin tests.online.CustomHandler
2019-01-22 10:51:40,522 [kafka-monitor] DEBUG: Connected to Redis in ActionHandler
2019-01-22 10:51:40,522 [kafka-monitor] DEBUG: Successfully loaded plugin tests.online.CustomHandler
2019-01-22 10:51:40,524 [kafka-monitor] DEBUG: Connected to Redis in StatsCollector Setup

I've manually checked both redis and kafka connections and know that they are able to communicate... I also see the topic created - but nothing passed to it (I created another consumer) I've also seen that a Consumer object is created.... Any ideas?

madisonb commented 5 years ago

Are you using containers to run your setup or just the scripts themselves? Perhaps the kafka-monitor cannot connect to Zookeeper on port 2181. If you can give me steps to reproduce via docker-compose (from the dev branch as the latest code is there) I can assist further, otherwise I am going to close this issue.

danmsf commented 5 years ago

Just the scripts, I also tested zookeeper via telnet and was able to communicate, I'll see if I can docker it

On Tue, Jan 22, 2019, 4:50 PM Madison Bahmer <notifications@github.com wrote:

Are you using containers to run your setup or just the scripts themselves? Perhaps the kafka-monitor cannot connect to Zookeeper on port 2181. If you can give me steps to reproduce via docker-compose (from the dev branch as the latest code is there) I can assist further

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/istresearch/scrapy-cluster/issues/213#issuecomment-456426607, or mute the thread https://github.com/notifications/unsubscribe-auth/ASwvtMlwiCYUHmvX1KUmHPbBo5Zb6FbQks5vFyUggaJpZM4aMgeE .

madisonb commented 5 years ago

Closing due to inactivity