Some jobs remain in processing state forever

dibbhatt / kafka-spark-consumer

High Performance Kafka Connector for Spark Streaming.Supports Multi Topic Fetch, Kafka Security. Reliable offset management in Zookeeper. No Data-loss. No dependency on HDFS and WAL. In-built PID rate controller. Support Message Handler . Offset Lag checker.

Apache License 2.0

635 stars 318 forks source link

Some jobs remain in processing state forever #26

Closed sorabh89 closed 8 years ago

sorabh89 commented 9 years ago

Hi Dibbhatt,

My application is running normally except that the UI shows some jobs remain in processing state forever.

Following is the screenshot of the same:

screenshot from 2015-09-22 17 55 25

Regards, Sorabh

dibbhatt commented 9 years ago

Hi,

I can see there are large scheduling delay of 8min or so. I guess all these jobs are still queued ? If you are using latest version of consumer , you can set props.put("consumer.backpressure.enabled", "true"); to see if that helps your scheduling delay and back-pressure issue .

Dibyendu

dibbhatt commented 9 years ago

hi @sorabh89 . this issue is still there once you use the consumer.backpressure.enabled flag ?

dibbhatt commented 9 years ago

Hi @sorabh89

I have created a JIRA to track the progress of contributing back this project to Apache Spark.

https://issues.apache.org/jira/browse/SPARK-11045

This project is now presently in spark-packages and I believe this is the correct time to contribute it to Apache Spark Project and give better options to larger community around Kafka Connectivity for Spark Streaming.

kindly Vote for this JIRA.