Problem:
We've created the diagram for the high level architecture, but it doesn't contain the details about the logical flow to actually Extract, Load and Transform data to be used by the Web App.
Create a diagram that shows the logical flow for the ELT cron job and how data is accessed on the Web App.
Data storage consists of three layers (all under the same Azure Storage Account, but in different Azure Blob Containers):
Bronze: raw data from source systems
Silver: filtered, cleaned and augmented data
Gold: aggregated data ready for reporting
The ELT cron job consists of three sequential tasks:
Extract:
Export data from Azure Cost Management REST API
Write raw data into Bronze layer
Load:
Read data from Bronze layer
Apply schema into dataset
Write data into Silver layer
Transform:
Reads data from Silver layer
Run transformations
Write data into Gold layer
The ELT cron job has an Azure Service Principal as Managed Identity, which has Storage Blob Data Contributor role on all blob containers
The Web App has an Azure Service Principal as Managed Identity, which has Storage Blob Data Reader role on the Gold blob container
Problem: We've created the diagram for the high level architecture, but it doesn't contain the details about the logical flow to actually Extract, Load and Transform data to be used by the Web App.
Create a diagram that shows the logical flow for the ELT cron job and how data is accessed on the Web App.