Azure-Samples / modern-data-warehouse-dataops

DataOps for the Modern Data Warehouse on Microsoft Azure. https://aka.ms/mdw-dataops.
MIT License
570 stars 451 forks source link

Add functionality for "Config files upload to lakehouse" as part of CI/CD #710

Open promisinganuj opened 4 months ago

promisinganuj commented 4 months ago

As part of Fabric workspace integration with git, only the lakehouse metadata gets source-controlled and promoted.

To use config-driven data pipelines (CDDP), the config files are typically required to be uploaded in the "Files" section of the lakehouse. So, there has to be a mechanism to:

Optional Functionality

promisinganuj commented 4 months ago

The script has been added here - https://github.com/Azure-Samples/modern-data-warehouse-dataops/blob/main/single_tech_samples/fabric/fabric_ci_cd/scripts/lakehouse-file-upload/upload-file-to-lakehouse.py

Now, the CI/CD pipelines need to be updated to include this step.