refactor extract function to extract only changes in the data

makikooni / DE_Engineering_Project

The project creates applications to Extract, Transform and Load data from a operational database for our client Terriffic Totes. The data is archived in a data lake, and is made available in a remodelled OLAP data warehouse hosted in AWS.

0 stars 2 forks source link

refactor extract function to extract only changes in the data #97

Closed atif-hussain-khan closed 1 year ago

atif-hussain-khan commented 1 year ago

Refactor extract function to extract only changes in the data

if no data in bucket, extract all data
if data exists. exract changed data
store all data and changes indefinitley in s3
keep a track of latest data for the transform lambda