search

keboola / db-extractor-mssql

MIT License
1 stars 2 forks source link

Add incremental fetching using change tracking #156

Open toma10 opened 4 years ago

toma10 commented 4 years ago

Hi.

At my company, I had to add change tracking option to this extractor. Feel free to merge it, change it or delete it.

pivnicek commented 4 years ago

Wow, thanks for sharing this with us Tomas! We'll take a look and let you know.

toma10 commented 4 years ago

Hi. I fixed some bugs and change some behavior. There was a requirement for the first run to fully load the table. In addition, the use of incrementalFetchingLimit was forbidden because problems could arise if multiple records were inserted at once. Change tracking in this case assigns the same sys_change_version to the inserted rows, which could mean that some new records might not be exported. In addition, the behavior will still be modified so that it does not have to specify an incremntalFetchingColumn, because it is always the primary key + there will be support for multiple primary keys. It is already more custom solution, so it is a question whether it belongs to this extractor. Maybe we could generalize it for this extractor.