[D5] - Consolidate Data into Denorm Table

[ydaponte]

As a Data Engineer, I want to consolidate all relevant data into a denorm table so that it can be used for scoring and tagging stale content.

Acceptance Criteria:

• Design a schema for the denorm table that includes relevant fields from different data sources.
• Implement data pipelines to extract, transform, and load (ETL) data into the denorm table.
• Ensure data integrity and consistency in the denorm table.

Potential Tech Stack to use:

• Denormalized data to be stored in Warehouse/Lakehouse (same as previous user story). Fabric Pipelines or Notebooks for data movement/processing between raw data and scored data.

[SQLZane]

Was able to build the table that we are pulling data into. We've also finalized the MERGE process in Python so that each data flow can each maintain it's own natural key within one flat table. This should assure that we can all put this logs in a central location without overwriting each others data. The notebook for table creation is DenormalizedItemCollection.