LOAD: write load lambda

[tricia-mcmillan]

extraction_lambda()

Depends on

• trigger from couldwatch schedule

inputs (info required):

• data from processed zone

output(s)

• logging (process and errors) + error handling

Function calls the following utility functions:

• get data from ingestion zone
• check for updates? persist timestamps somewhere?
• load/send data to warehouse

Exceptions:

[makikooni]

Awaiting utilities to proceed further.

[makikooni]

FULL LOAD LAMBDA workflow.

• Access the s3 bucket
• Compare timestamp with current one (check if new one was added)
  • IF NOT : Return
  • IF YES:
    • If new rows - run Lambda ✅
    • If update: ( how do we know update was added to update)
      • For each new data update:
        • SQL string for update:
          • Find the row that contains updated data
          • Replace the whole row with new row containing that data (do we need to copy the whole row first? or are we gonna upload whole rows?)

TEST 1) Check if it is an update or adding new rows.