search

hortonworks-spark / spark-atlas-connector

A Spark Atlas connector to track data lineage in Apache Atlas
Apache License 2.0
264 stars 149 forks source link

Flaky test failures on streaming query suite #281

Open HeartSaVioR opened 4 years ago

HeartSaVioR commented 4 years ago 
- file source to kafka sink - micro-batch query *** FAILED *** (1 second, 301 milliseconds)
  java.lang.AssertionError: assertion failed: 
  Set(
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-9.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-8.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-0.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-3.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-4.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-5.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-7.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-1.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-10.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-6.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-2.txt) 

  is not a subset of 

  Set(
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-0.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-3.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-4.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-5.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-1.txt, 
  file:///spark-atlas-connector-hortonworks/spark-atlas-connector/target/tmp/spark-atlas-streaming-files-temp8973477114977388256/src/file-2.txt
  )
  at scala.Predef$.assert(Predef.scala:170)
  at com.hortonworks.spark.atlas.TestUtils$.assertSubsetOf(TestUtils.scala:60)
  at com.hortonworks.spark.atlas.sql.SparkExecutionPlanProcessorForStreamingQuerySuite$$anonfun$12$$anonfun$apply$13.apply(SparkExecutionPlanProcessorForStreamingQuerySuite.scala:357)
  at com.hortonworks.spark.atlas.sql.SparkExecutionPlanProcessorForStreamingQuerySuite$$anonfun$12$$anonfun$apply$13.apply(SparkExecutionPlanProcessorForStreamingQuerySuite.scala:354)
  at com.hortonworks.spark.atlas.sql.SparkExecutionPlanProcessorForStreamingQuerySuite.com$hortonworks$spark$atlas$sql$SparkExecutionPlanProcessorForStreamingQuerySuite$$assertEntitySparkProcessType(SparkExecutionPlanProcessorForStreamingQuerySuite.scala:694)
  at com.hortonworks.spark.atlas.sql.SparkExecutionPlanProcessorForStreamingQuerySuite$$anonfun$12.apply(SparkExecutionPlanProcessorForStreamingQuerySuite.scala:354)
  at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  ...
- file & kafka source to kafka sink - micro-batch query *** FAILED *** (15 seconds, 589 milliseconds)
  3 did not equal 11 (FsEntityValidator.scala:42)