search

Azure-Player / azure.datafactory.tools

Tools for deploying Data Factory (v2) in Microsoft Azure
https://azureplayer.net/adftools
MIT License
216 stars 69 forks source link

ADF Deployment - Incremental Deployment: (404) Not Found. #401

Closed NowinskiK closed 2 months ago

NowinskiK commented 3 months ago

Discussed in https://github.com/Azure-Player/azure.datafactory.tools/discussions/400

Originally posted by **NobleNooblet** July 15, 2024 I've been looking into different ways to deploy ADF infrastructure to different environments and I came across this tool which is super cool! I've created a simple setup to test the deployment process and I've picked up an issue when using the "Incremental Deployment (new)" functionality. I'll try and explain my process as simple as possible. # My setup: DevOps Repo - let's call it DevOps-master Data Factory in Dev - let's call it adf-dev (linked to DevOps-master) Data Factory in Test - let's call it adf-test (which is my target deployment) I've created a simple release pipeline that triggers when any changes are pushed to DevOps-master and it deploys the changes to adf-test. ## Test scenario 1: Incremental Deployment set to False Step 1: Create a new pipeline: pipeline1 Step 2: Add an activity setVariable and populate it with dummy data Step 3: Save changes (which pushes the changes to DevOps-master) Result: Release is triggered Release completed successfully Confirmed that adf-test contains the changes made in adf-dev ## Test scenario 2: Repeat Test scenario 1 with Incremental Deployment set to True Result: Release is triggered Release failed with error: ``` 2024-07-15T02:56:22.6997385Z Publish options are provided. 2024-07-15T02:56:22.7006802Z STEP: Verifying whether ADF exists... 2024-07-15T02:56:23.6038710Z ##[warning]You're using Az.DataFactory version 1.18.4. The latest version of Az.DataFactory is 1.18.5. Upgrade your Az modules using the following commands: Update-Module Az.* -WhatIf -- Simulate updating your Az modules. Update-Module Az.* -- Update your Az modules. 2024-07-15T02:56:23.6275981Z Azure Data Factory exists. 2024-07-15T02:56:23.6304705Z Loading Deployment State from ADF... 2024-07-15T02:56:24.4575155Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop 2024-07-15T02:56:24.4931467Z ##[command]Clear-AzContext -Scope Process -ErrorAction Stop 2024-07-15T02:56:24.6031159Z ##[error]The remote server returned an error: (404) Not Found. 2024-07-15T02:56:24.6672637Z ##[section]Finishing: Publish ADF adf-test from JSON files ``` No changes reflecting in adf-test I've played around a bit and managed to find a workaround. Here's what I've done. Navigated to adf-test -> created a global parameter of type Object called adftools_deployment_state with value {} and published the change. I repeated test scenario 2 and this time it completed successfully, and I can confirm that adftools_deployment_state updated with hash values. Here's my take on it: To me it seems that the incremental process fails if you have no global parameter(s) set up. Adding the global parameter adftools_deployment_state manually resolved my issue, and I can make use of the incremental function going forward. Is this potentially a bug or am I understanding the use of "incremental Deployment" incorrectly? Thank you for your time
NowinskiK commented 2 months ago

Please look at #374. The incremental deployment feature will no longer work with GP (Global Parameter) in ADF. The Incremental Deployment State will be saved into Azure Storage as JSON file.