Not able to see recommendation for StreamingLens in Logs?

[rpatid10]

Hi,

I am trying to Implement StreamingLens In My existing Streaming Application. My application Is working fine and its loading data from one kafka topic to anothe kafka topic. But in ambari I am not able to see StreamingLens Reports, when I did this for batch application using sparklence i could see the logs generated by sparklense with all the resources information but same report I am not able to see for streaming application can someone suggest if I need to do additional code or where should I check the reports which should generate by StreamingLens.

My Sample code

      class SparkStreamingLens(spark: SparkSession, options: RequestBuilder)
      object SparkStreamingLens {
      def main(args: Array[String]): Unit = {
      println(" Spark Parameters are :")
      val igniteDetails = args(0)  
      val applicationName = args(1)
      val argumentTable = args(2)
      // options.addParameter("streamingLens.reporter.intervalMinutes", "1")
      val spark = SparkSession
      .builder()
      .appName(applicationName)
      .getOrCreate()
      val streamingLens = new SparkStreamingLens(spark, options)
      // Remaining Code to Read from Kafka and write Into Kafka(Streaming Data)
      }
      }

  spark-submit Command:

  spark-submit \
  --verbose \
  --name SparkStreamingLens \
  --num-executors 1  \
  --conf streamingLens.reporter.intervalMinutes=1  \
  --jars /home/abc/jars/spark-streaminglens_2.11-0.5.3.jar,\
 /home/abc/jars/kafka-clients-0.10.2.1.jar,\
  --master yarn \
  --deploy-mode cluster \
  --driver-cores 1  --driver-memory 2G  --executor-cores 1  --executor-memory 2G \
  --supervise --class com.data.datalake.SparkStreamingLens \
 /home/abc/jar/SparkStreamingLens-spark-utility_2.11-1.0.jar  \
 "jdbc:ignite:thin://00.000.00.00:00000;distributedJoins=true;user=aaaaaa;password=aaaaaaa;"  \
 SparkStreamingLens \
 argumentTable

[rpatid10]

@abhishekd0907 Kindly suggest.

[abhishekd0907]

Can you share your Spark driver logs?

You need to add below line to activate it

val streamingLens = new StreamingLens(sparkSession, options)

[rpatid10]

Can you share your Spark driver logs?

You need to add below line to activate it

val streamingLens = new StreamingLens(sparkSession, options)

Yes I have added this line @abhishekd0907 val streamingLens = new SparkStreamingLens(spark, options)

[rpatid10]

@abhishekd0907 Thanks a lot . It worked and I am able to see the logs. 21/09/29 17:38:42 INFO QueryInsightsManager:																		StreamingLens Inisights																									BatchId: 344 Analysis Time: 00s 000ms Expected Micro Batch SLA: 120s 000ms Batch Running Time: 00s 000ms Critical Time: 00s 000ms Streaming Query State: NONEWBATCHES