CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Optimise sql server query to detect highest timestamp for table correctly #104

Closed jamesfielder closed 4 years ago

jamesfielder commented 4 years ago

Description

Change the sql server temporal extractor to select the upper bound timestamp bound from the table, fixing problems where the upper bound would be '9999-12-31 23:59:59.0000000' rather than '9999-12-31 23:59:59.9999999' when casting Datetime2(0) to Datetime2(7).

Fixes # (issue)

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Tested against the existing database tests and manually by running sql against the db here at cox auto.