Ingestion phase crashes when eventTime column is not specified or when contains null values
Exception in thread "main" java.lang.NullPointerException
at dev.kamu.engine.spark.ingest.Ingest.getLastWatermark(Ingest.scala:336)
at dev.kamu.engine.spark.ingest.Ingest.ingest(Ingest.scala:101)
at dev.kamu.engine.spark.ingest.Ingest.ingest(Ingest.scala:42)
at dev.kamu.engine.spark.ingest.IngestApp$.main(IngestApp.scala:45)
at dev.kamu.engine.spark.ingest.IngestApp.main(IngestApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
sergiimk
commented
3 years ago
Ingestion phase crashes when eventTime column is not specified or when contains null values