Azure-Player / azure.datafactory.tools

Tools for deploying Data Factory (v2) in Microsoft Azure
https://azureplayer.net/adftools
MIT License
207 stars 69 forks source link

More efficient testing by deploying into subfolder -- multi deployment into one ADF instance #361

Closed fmms closed 11 months ago

fmms commented 11 months ago

Is your feature request related to a problem? Please describe. I am using Azure DevOps to fully test the data warehouse before merging into onto the main branch.

This works very well for a database where another database is created on the same SQL Server and the filled with sqlpackage. It works as well with SSIS where we just create a parallel folder structure on the DEV instance of Azure IS. SSAS/Azure AS allow as well to just create another model on the existing instance.

However, for ADF a new resource has to be created, in case one is not just using native ADF but as well integrated Azure IS with custom components deployed into that platform it takes more than 10 minutes just to setup and boot. This is a very lengthy process to validate PRs.

Describe the solution you'd like I imagine giving a prefix to objects in ADF deployment. As a result all pipelines would go into a specific subfolder, their files would all be prefixed with a certain string. Moreover, all datasets, linked services would be prefiixed as well and all references to those would be adjusted.

This would mean that I could deploy multiple instances of my ADF project to the same ADF and save an enourmous amount of time creating and destroying azure resources.

Describe alternatives you've considered Create new ADF instances including SSIS-IR on-demand, automatically leverage linked SHIR for onPrem connectivity.

Additional context Add any other context or screenshots about the feature request here.

Does this seem like a reasonable idea? Do others see the need as well?

NowinskiK commented 11 months ago

This is edge case scenario and would require loads of effort to implement. You can write your own PowerShell script to rename all objects (and files) locally before kick off the deployment.