CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Fix the SQL Server Temporal Extractor when extracting from an empty table #105

Closed jamesfielder closed 4 years ago

jamesfielder commented 4 years ago

Description

After the previous changes we select the max ts from a history table to determine what the max timestamp should be. However, when the table is empty this query returns null, which causes repeated extractions to fail. This change instead has the value default to the set max timestamp, instead of returning null in this case.

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Added a new test and tested manually against both the running test db and a db here at cox auto.