Azure-Samples / modern-data-warehouse-dataops

DataOps for Microsoft Data Platform technologies. https://aka.ms/dataops-repo

MIT License

590 stars 462 forks source link

feat: update diagram #863

Closed devlace closed 1 day ago

devlace commented 1 day ago

Type of PR

Documentation changes

Purpose

Update Fabric E2E Architecture

Does this introduce a breaking change? If yes, details on what can break

Author pre-publish checklist

[ ] Added test to prove my fix is effective or new feature works
[X] No PII in logs
[ ] Made corresponding changes to the documentation

Validation steps

Issues Closed or Referenced

References #788

devlace commented 1 day ago

Using Copy + SQL Proc instead of Spark notebooks to copy data into Warehouse. Essentially, Copy data into "interim" tables in the warehouse, then call a Stored Proc to transform into Star Schema.

Alternatives tried:

Spark Notebook - no support for landing data directly to a warehouse
TSQL Notebook - currently no support for notebook parameters. No support for Git and Deployment pipelines.