Azure-Samples / modern-data-warehouse-dataops

DataOps for Microsoft Data Platform technologies. https://aka.ms/dataops-repo
MIT License
591 stars 463 forks source link

Add functionality for "Config files upload to lakehouse" as part of CI/CD #710

Closed promisinganuj closed 3 weeks ago

promisinganuj commented 6 months ago

As part of Fabric workspace integration with git, only the lakehouse metadata gets source-controlled and promoted.

To use config-driven data pipelines (CDDP), the config files are typically required to be uploaded in the "Files" section of the lakehouse. So, there has to be a mechanism to:

Optional Functionality

promisinganuj commented 6 months ago

The script has been added here - https://github.com/Azure-Samples/modern-data-warehouse-dataops/blob/main/single_tech_samples/fabric/fabric_ci_cd/scripts/lakehouse-file-upload/upload-file-to-lakehouse.py

Now, the CI/CD pipelines need to be updated to include this step.

devlace commented 3 weeks ago

Closing to be succeeded by new issue to move script to utility folder