keboola / db-extractor-mssql

MIT License
1 stars 2 forks source link

Incremental extraction fails when using datetime column and there are other columns with null in the last extracted row #119

Open tomasfejfar opened 5 years ago

tomasfejfar commented 5 years ago

Because the datetime value is converted from it's original form (mili-/microseconds stripped) we can't really tell which of the last rows was it based just on the value. So we check all the other columns values to see which one of the matching datetimes match the other fields also.

Problem is that with BCP all NULLs are converted to empty strings. Because of that when fetching the last record nothing is matched:

SELECT [ SMALLDATETIME]
FROM [dbo].[auto Increment TIMESTAMP]
WHERE [_Weir%d I-D] = 8 AND
 [Weir%d Na-me] = 'william' AND
 [someInteger] = '' AND
 [someDecimal] = '' AND
 [TYPE] = '' AND
 [ SMALLDATETIME] = "2012-01-10 10:55:00" AND
 CONVERT(DATETIME2(0), [ DATETIME]) = "2019-05-29 18:07:58";

Solution:

Because we are currently checking the last fetched row value using >= we can take the last fetched datetime, that might have been rounded to one second up. The only problem we need to solve is how to substract the possible rounded second. We could either find a solution in the database or we can substract one second from the result in PHP.

tomasfejfar commented 5 years ago

https://stackoverflow.com/a/3287213/112000