makikooni / DE_Engineering_Project

The project creates applications to Extract, Transform and Load data from a operational database for our client Terriffic Totes. The data is archived in a data lake, and is made available in a remodelled OLAP data warehouse hosted in AWS.

0 stars 2 forks source link

EXTRACT: create lambda extraction resource #13

Open dabruge opened 1 year ago

dabruge commented 1 year ago

extraction_lambda()

Depends on

trigger via schedule (cloudwatch)

inputs (info required):

needs to know db data is held

output(s)

logging (process and errors) + error handling

Function calls the following utility functions:

check for changes in RDBMS - if no changes: log no changes
query and extract the data
send data to ingestion zone

Exceptions: