qubole / streaminglens

Qubole Streaminglens tool for tuning Spark Structured Streaming Pipelines
http://www.qubole.com
Apache License 2.0
17 stars 5 forks source link

Streaming Lens failed key not found: BatchDescription #6

Closed rpatid10 closed 2 years ago

rpatid10 commented 2 years ago

Can Someone suggest what is the issue here I am getting this logs while executing spark-structure-streaming Application with StreamingLence.Its Not generating Any Recommendation and state.

21/10/01 11:30:07 WARN QueryInsightsManager: Streaming Lens failed key not found: BatchDescription(62618e99-5175-4106-90f1-0b2ef0b49fda,2) 21/10/01 11:30:07 INFO QueryInsightsManager: Max retries reached. Attempting to stop StreamingLens 21/10/01 11:30:07 INFO QueryInsightsManager: Successfully shutdown StreamingLens

Spark Submit Command.

spark Submit Command:

spark-submit --verbose --name SparkStreamingLens --num-executors 1 --conf streamingLens.reporter.intervalMinutes=1 --jars /home/abc/jars/spark-streaminglens_2.11-0.5.3.jar, /home/abc/jars/kafka-clients-0.10.2.1.jar, --master yarn --deploy-mode cluster --driver-cores 1 --driver-memory 2G --executor-cores 1 --executor-memory 2G --supervise --class com.data.datalake.SparkStreamingLens /home/abc/jar/SparkStreamingLens-spark-utility_2.11-1.0.jar

Kindly help.