search

Stratio / spark-rabbitmq

RabbitMQ Spark Streaming receiver
Apache License 2.0
208 stars 84 forks source link

Consumers/Executors hanging #103

Open kaveh-hariri opened 7 years ago

kaveh-hariri commented 7 years ago

Hi,

I'm using the distributed approach, 1 queue, 10 consumers/containers. After some time, a consumer will "hang" in the sense that the streaming window closes and moves on, but the container from the previous window never finishes. I've noticed that this causes a big drag in performance.

In the picture below there should be 10 consumers/containers, but RabbitMQ shows 12. The prefetch count is at 100, and you see the messages ready and unacked messages at 200. This is the new floor as 2 of the consumers/containers are hung up. If I kill the job completely and restart, then these 2 hung consumers go away. Any idea what may be causing this? I've also pasted my configurations below.

image

image

nelsou commented 7 years ago

+1 got it too. I made a quick fix with a Thread that force to close "unused connection".

compae commented 7 years ago

Hi,

@nelsou could you make one PR with the bugfixing?

yenki commented 7 years ago

+1 I'm having this problem also