I am using the incremental refresh on a date field but when the connector passes dates to Tableau, it strips out the milliseconds accuracy.
So when you make a incremental refresh, it might include duplicate records if there are multiple on the same second. Example:
1) Existing records:
2017-07-07T01:01:01.200Z Record 1
Incremental refresh from an earlier date will include this record correctly.
2) Existing records, a new record (Record 2) is added:
2017-07-07T01:01:01.200Z Record 1
2017-07-07T01:01:01.300Z Record 2
Incremental refresh will be done from this date: 2017-07-07T01:01:01.000Z
But it will exclude the first record (see here) so this is expected behaviour and only Record 2 will be added.
3) Existing records, a new record (Record 3) is added:
2017-07-07T01:01:01.200Z Record 1
2017-07-07T01:01:01.300Z Record 2
2017-07-07T01:01:01.400Z Record 3
Incremental refresh will be done from this date: 2017-07-07T01:01:01.000Z because the milliseconds are not saved.
It will exclude the first record but not the second one, so record 2 will be duplicated in the Tableau extract.
My workaround for now is to force all dates to be integers and making the ElasticSearch query with a timestamp instead of a date. So I don't have a proper pull request to offer because this removes the Date parsing from the connector.
I am using the incremental refresh on a date field but when the connector passes dates to Tableau, it strips out the milliseconds accuracy. So when you make a incremental refresh, it might include duplicate records if there are multiple on the same second. Example:
1) Existing records: 2017-07-07T01:01:01.200Z Record 1 Incremental refresh from an earlier date will include this record correctly.
2) Existing records, a new record (Record 2) is added: 2017-07-07T01:01:01.200Z Record 1 2017-07-07T01:01:01.300Z Record 2 Incremental refresh will be done from this date: 2017-07-07T01:01:01.000Z But it will exclude the first record (see here) so this is expected behaviour and only Record 2 will be added.
3) Existing records, a new record (Record 3) is added: 2017-07-07T01:01:01.200Z Record 1 2017-07-07T01:01:01.300Z Record 2 2017-07-07T01:01:01.400Z Record 3 Incremental refresh will be done from this date: 2017-07-07T01:01:01.000Z because the milliseconds are not saved. It will exclude the first record but not the second one, so record 2 will be duplicated in the Tableau extract.
My workaround for now is to force all dates to be integers and making the ElasticSearch query with a timestamp instead of a date. So I don't have a proper pull request to offer because this removes the Date parsing from the connector.