search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.17k stars 323 forks source link

Update Releases, Docker compose Issues #238

Closed MrMoon closed 3 years ago

MrMoon commented 3 years ago

There a some issues in the docker compose file in the release ( Scrapy Cluster 1.2.1 ), which is linked in the documentation, I spend about 2 hours trying to find why my kafka cluster won't start or even pass the test.

Here is one of the errors, I've grabbed it from the log [2020-10-23 14:29:08,832] WARN [Controller id=1, targetBrokerId=1] Error connecting to node broker:29092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient) java.net.UnknownHostException: broker: Temporary failure in name resolution at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) at java.net.InetAddress.getAllByName0(InetAddress.java:1277) at java.net.InetAddress.getAllByName(InetAddress.java:1193) at java.net.InetAddress.getAllByName(InetAddress.java:1127) at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403) at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363) at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151) at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:962) at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294) at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:65) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:297) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:251) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

I ended up cloning the repository and it worked like a charm.

anthony9981 commented 3 years ago

I wish I read this before. I spend two more hours to resolve this issue in 1.2.1 release version.

madisonb commented 3 years ago

I'm not clear what the fix is here, but I am closing this as you seem to have resolved it on your own.