Incase of triggers renaming, new triggers getting created but original triggers not getting deleted with selective deployment of Azure Data Factory

[TejoGowthu-Zoetis]

Describe the bug We are using selective deployment of objects using include exclude rules in file. Observed that if we rename the existing trigger in JSON file and try to publish to ADF then its creating renamed trigger which is expected but still keeping the original trigger in stopped state instead of deleting it (which is not expected). But if we rename pipelines, its deleting old ones and creating with new name which is expected. Seems like this issue observed with only triggers.

To Reproduce Your script here: ./.github/scripts/deploy.ps1 -SubscriptionName "SubName1" -ResourceGroupName "Test-RG1" -DataFactoryName "Test-DF1" -Location "eastus2" -RootFolder ""/home/runner/work/adf-dev/adf-dev"" -Stage "dev" -DeleteNotInSource:$true -IncludeExcludeFile "./.github/publish-includeexclude-objects.txt"

deploy.ps1 script: param ( [parameter(Mandatory = $true)] [String] $SubscriptionName, [parameter(Mandatory = $true)] [String] $ResourceGroupName, [parameter(Mandatory = $true)] [String] $DataFactoryName, [parameter(Mandatory = $true)] [String] $Location, [parameter(Mandatory = $true)] [String] $RootFolder, [switch] $DeleteNotInSource = $false, [String] $Stage = "dev", [parameter(Mandatory = $true)] [String] $IncludeExcludeFilePath )

Install-Module -Name Az.DataFactory -Scope CurrentUser -Force # required by azure.datafactory.tools

Install-Module -Name azure.datafactory.tools -Scope CurrentUser -Force Import-Module -Name azure.datafactory.tools

Run TestAdfCode function to validate JSON

Test-AdfCode -RootFolder "$RootFolder"

Set deploy parameters

$opt = New-AdfPublishOption -FilterFilePath $IncludeExcludeFilePath $opt.DeleteNotInSource = $DeleteNotInSource $opt.StopStartTriggers = $true $opt.DeployGlobalParams = $false $opt.CreateNewInstance = $false

Deploy to Data Factory

Publish-AdfV2FromJson -RootFolder "$RootFolder" -ResourceGroupName "$ResourceGroupName" -DataFactoryName "$DataFactoryName" -Location "$Location" -Stage $Stage -Option $opt

Objects publish-includeexclude-objects.txt file: +pipeline.test_selectivemigration_pipeline1 +trigger.test_selectivemigration_trigger1

Expected behaviour If triggers are renamed then it should create new trigger as per renaming and should delete the one with old name instead of keeping it in stopped state.

Screenshots If applicable, add screenshots to help explain your problem.

Version of PowerShell module of: azure/powershell@v1 Importing module > 1.10.0

Provide all files to reproduce the issue

• Attach at least log file (you can wipe out any potentially sensitive info)
• Attach files of objects which are involved in the process

Detailed Deployment Log: STEP: Reading Azure Data Factory from JSON files... IntegrationRuntimes: 1 object(s) loaded. LinkedServices: 10 object(s) loaded. Pipelines: 79 object(s) loaded. DataSets: 27 object(s) loaded. DataFlows: 0 object(s) loaded. Triggers: 14 object(s) loaded. Managed VNet: 0 object(s) loaded. Managed Private Endpoints: 0 object(s) loaded. Factories: 1 object(s) loaded.

Number of objects marked as to be deployed: 1/132

• [trigger].[test_selectivemigration_trigger3]

STEP: Stopping triggers... Getting triggers... The number of triggers to stop: 12 (out of 17) Stopping deployed triggers:

• Disabling trigger: TEST1 trigger
• Disabling trigger: TEST3 trigger
• Disabling trigger: TEST4 trigger
• Disabling trigger: TEST5 trigger
• Disabling trigger: TEST6 trigger
• Disabling trigger: TEST2 trigger
• Disabling trigger: TEST7 trigger
• Disabling trigger: TEST8 trigger
• Disabling trigger: TEST9 trigger
• Disabling trigger: TEST10 trigger
• Disabling trigger: TEST11 trigger
• Disabling trigger: TEST12 trigger Complete stopping deployed triggers

STEP: Deployment of all ADF objects... Deployment of Global Parameters will be skipped as publish option 'DeployGlobalParams' = false Start deploying object: [trigger].[test_selectivemigration_trigger3] (1 dependency/ies) Finished deploying object: [trigger].[test_selectivemigration_trigger3]

STEP: Deleting objects not in source ... Azure Data Factory (instance) loaded. DataSets: 27 object(s) loaded. IntegrationRuntimes: 1 object(s) loaded. LinkedServices: 10 object(s) loaded. Pipelines: 79 object(s) loaded. DataFlows: 0 object(s) loaded. Triggers: 18 object(s) loaded.

STEP: Starting all triggers... The number of triggers to start: 12

• Enabling trigger: TEST1 trigger
• Enabling trigger: TEST3 trigger
• Enabling trigger: TEST4 trigger
• Enabling trigger: TEST5 trigger
• Enabling trigger: TEST6 trigger
• Enabling trigger: TEST7 trigger
• Enabling trigger: TEST8 trigger
• Enabling trigger: TEST9 trigger
• Enabling trigger: TEST10 trigger
• Enabling trigger: TEST11 trigger
• Enabling trigger: TEST2 trigger
• Enabling trigger: TEST12 trigger

[TejoGowthu-Zoetis]

Hi @NowinskiK, Can you please let me know if you have any update on this issue.

[NowinskiK]

Not really. This case requires preparation of whole PS script to be able to test it. If you can prepare such script and attach all required files to prove this is really happens and it is repeatable - that can help me to investigate that over reduce amount of time required.

[TejoGowthu-Zoetis]

I have uploaded requested files, please rename them as mentioned below. Rename "deploy powershell script.txt" to "deploy.ps1" Rename "test_selectivemigration_pipeline1 JSON.txt" to "test_selectivemigration_pipeline1.json" Rename "test_selectivemigration_trigger1 JSON.txt" to "test_selectivemigration_trigger1.json"

Command to Run: ./.github/scripts/deploy.ps1 -SubscriptionName "SubName1" -ResourceGroupName "Test-RG1" -DataFactoryName "Test-DF1" -Location "eastus2" -RootFolder ""/home/runner/work/adf-dev/adf-dev"" -Stage "dev" -DeleteNotInSource:$true -IncludeExcludeFile "./.github/publish-includeexclude-objects.txt"

Uploaded below files. publish-includeexclude-objects.txt deploy powershell script.txt test_selectivemigration_pipeline1 JSON.txt test_selectivemigration_trigger1 JSON.txt

[TejoGowthu-Zoetis]

Hi @NowinskiK, As requested I have uploaded the PS script. Could you please let me know if you have any update.

[NowinskiK]

Thanks for attaching the files, including the script. Unfortunately, I can't see how that can help to reproduce the issue. The script contains only one execution of Publish-AdfV2FromJson, do it will create 1 trigger and 1 pipeline; that's it. Of course, after the first run I can rename the trigger and rerun the script again - I just need to be sure that you tested the feature in exactly that way?

[TejoGowthu-Zoetis]

@NowinskiK Yes we tested with only one pipeline and trigger only, if we rename any trigger then ADF not deleting the original trigger and we end up having 2 triggers. Please let me know if you have any more questions.

[NowinskiK]

I prepared appropriate and full test script, it is in the repo now: https://github.com/SQLPlayer/azure.datafactory.tools/tree/develop/adhoc/issue-223

Additionally, I recorded quick video to show you, that the scenario you described, works: https://youtu.be/B_IgLfAdLgg

[NowinskiK]

Probably I know what scenario you're referring to. Yes, it will omit deleting renamed trigger if it's excluded from the deployment. In such case you must set the option DoNotDeleteExcludedObjects = false, which means it will delete excluded objects.

Doc reference: Step: Deleting objects not in source