search

Azure-Player / azure.datafactory.devops

Azure DevOps Deployment Tasks for Azure Data Factory objects
MIT License
39 stars 21 forks source link

Triggers not re-enabling after deployment #91

Closed clpalmer closed 2 years ago

clpalmer commented 2 years ago

I'm seeing the same issue as was shown here: https://github.com/SQLPlayer/azure.datafactory.devops/issues/85

Not able to paste the entire log, but hopefully the below contains the relevant info.

First it did the usual verifying whether ADF exists, then reading JSON files, then replacing the properties with the env specific CSV. After that it stopped all 6 triggers, as expected:

===================================================================================
STEP: Stopping triggers...
Getting triggers...
The number of triggers to stop: 6 (out of 7)
Stopping deployed triggers:
- Disabling trigger: <trigger name redacted>
- Disabling trigger: <trigger name redacted>
- Disabling trigger: <trigger name redacted>
- Disabling trigger: <trigger name redacted>
- Disabling trigger: <trigger name redacted>
- Disabling trigger: <trigger name redacted>
Complete stopping deployed triggers
===================================================================================

Then it proceeded to deploy the ADF objects, added global params, deleted some removed objects, again as expected.

Then it went to re-enable the triggers (one trigger was deleted during the deployment, but it was not used and not one of the 6 stopped ones. Other triggers weren't changed) and found none to re-enable:

===================================================================================
STEP: Starting all triggers...
The number of triggers to start: 0
==============================================================================

Rest of the deployment, aside from trigger re-enable went as expected, including addition/removal/update of objects, linked services, triggers, etc.

clpalmer commented 2 years ago

@NowinskiK - Ran it again to see whether it was the removal of the old trigger that broke the re-start of the rest, but it doesn't appear to have. The same happened again. Here's a more complete log, with the sensitive bits redacted:

Starting: Deploy Data Factory
==============================================================================
Task         : Publish Azure Data Factory
Description  : Deploys entire ADF (v2) from JSON files to ADF instance (adftools)
Version      : 1.11.1070
Author       : Kamil Nowinski (SQLPlayer)
Help         : [More Information](https://github.com/SQLPlayer/azure.datafactory.devops)
==============================================================================
PowerShell: 5.1.20348.643 Desktop
Added TLS 1.2 in session.
Import-Module -Name C:\Modules\az_7.3.2\Az.Accounts\2.7.6\Az.Accounts.psd1 -Global
Clear-AzContext -Scope Process
Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
Connect-AzAccount -ServicePrincipal -Tenant <redacted> -Credential System.Management.Automation.PSCredential -Environment AzureCloud @processScope
 Set-AzContext -SubscriptionId <redacted> -TenantId <redacted>
======================================================================================
### azure.datafactory.tools                                       Version 0.95.001 ###
======================================================================================
Invoking Publish-AdfV2FromJson (https://github.com/SQLPlayer/azure.datafactory.tools)
with the following parameters:
======================================================================================
RootFolder:         D:\a\1\src\<redacted>
ResourceGroupName:  <redacted>
DataFactoryName:    <redacted>
Location:           <redacted>
Stage:              <redacted>
Options provided:   True
Publishing method:  AzResource
Is Dry Run?:        False
======================================================================================
Publish options are provided.
STEP: Verifying whether ADF exists...
Azure Data Factory exists.
===================================================================================
STEP: Reading Azure Data Factory from JSON files...
IntegrationRuntimes: 0 object(s) loaded.
LinkedServices: 4 object(s) loaded.
Pipelines: 10 object(s) loaded.
DataSets: 0 object(s) loaded.
DataFlows: 0 object(s) loaded.
Triggers: 6 object(s) loaded.
Managed VNet: 0 object(s) loaded.
Managed Private Endpoints: 0 object(s) loaded.
Factories: 1 object(s) loaded.
# Number of objects marked as to be deployed: 21/21
- [linkedService].[<redacted>]
- [linkedService].[<redacted>]
- [linkedService].[<redacted>]
- [linkedService].[<redacted>]
- [pipeline].[<redacted>]
- [pipeline].[<redacted>]
- [pipeline].[<redacted>]
- [pipeline].[<redacted>]
- [pipeline].[<redacted>]
- [pipeline].[<redacted>]
- [pipeline].[<redacted>]
- [pipeline].[<redacted>]
- [pipeline].[<redacted>]
- [pipeline].[<redacted>]
- [trigger].[<redacted>]
- [trigger].[<redacted>]
- [trigger].[<redacted>]
- [trigger].[<redacted>]
- [trigger].[<redacted>]
- [trigger].[<redacted>]
- [factory].[<redacted>]
===================================================================================
STEP: Replacing all properties environment-related...
Config file:   D:\a\1\src\<redacted>\deployment\config-<redacted>.csv
Validation of config file completed.
*** Properties modification report ***

Name                           Value                                                                                   
----                           -----                                                                                   
Added                          0                                                                                       
Removed                        0                                                                                       
Updated                        48                                                                                      

===================================================================================
STEP: Stopping triggers...
Getting triggers...
The number of triggers to stop: 6 (out of 6)
Stopping deployed triggers:
- Disabling trigger: <redacted>
- Disabling trigger: <redacted>
- Disabling trigger: <redacted>
- Disabling trigger: <redacted>
- Disabling trigger: <redacted>
- Disabling trigger: <redacted>
Complete stopping deployed triggers
===================================================================================
STEP: Deployment of all ADF objects...
Start deploying object: [linkedService].[<redacted>] (1 dependency/ies)
Start deploying object: [linkedService].[<redacted>] (0 dependency/ies)
Finished deploying object: [linkedService].[<redacted>]
Finished deploying object: [linkedService].[<redacted>]
Start deploying object: [linkedService].[<redacted>] (1 dependency/ies)
Finished deploying object: [linkedService].[<redacted>]
Start deploying object: [linkedService].[<redacted>] (1 dependency/ies)
Finished deploying object: [linkedService].[<redacted>]
Start deploying object: [pipeline].[<redacted>] (1 dependency/ies)
Finished deploying object: [pipeline].[<redacted>]
Start deploying object: [pipeline].[<redacted>] (1 dependency/ies)
Finished deploying object: [pipeline].[<redacted>]
Start deploying object: [pipeline].[<redacted>] (1 dependency/ies)
Finished deploying object: [pipeline].[<redacted>]
Start deploying object: [pipeline].[<redacted>] (1 dependency/ies)
Finished deploying object: [pipeline].[<redacted>]
Start deploying object: [pipeline].[<redacted>] (2 dependency/ies)
Start deploying object: [pipeline].[<redacted>] (1 dependency/ies)
Finished deploying object: [pipeline].[<redacted>]
Finished deploying object: [pipeline].[<redacted>]
Start deploying object: [pipeline].[<redacted>] (1 dependency/ies)
Finished deploying object: [pipeline].[<redacted>]
Start deploying object: [pipeline].[<redacted>] (1 dependency/ies)
Finished deploying object: [pipeline].[<redacted>]
Start deploying object: [pipeline].[<redacted>] (1 dependency/ies)
Finished deploying object: [pipeline].[<redacted>]
Start deploying object: [pipeline].[<redacted>] (1 dependency/ies)
Finished deploying object: [pipeline].[<redacted>]
Start deploying object: [trigger].[<redacted>] (1 dependency/ies)
Finished deploying object: [trigger].[<redacted>]
Start deploying object: [trigger].[<redacted>] (1 dependency/ies)
Finished deploying object: [trigger].[<redacted>]
Start deploying object: [trigger].[<redacted>] (1 dependency/ies)
Finished deploying object: [trigger].[<redacted>]
Start deploying object: [trigger].[<redacted>] (2 dependency/ies)
Finished deploying object: [trigger].[<redacted>]
Start deploying object: [trigger].[<redacted>] (1 dependency/ies)
Finished deploying object: [trigger].[<redacted>]
Start deploying object: [trigger].[<redacted>] (1 dependency/ies)
Finished deploying object: [trigger].[<redacted>]
Start deploying object: [factory].[<redacted>] (0 dependency/ies)
Adding global parameter...
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
-  <redacted>
Update of 43 global parameters complete.
Finished deploying object: [factory].[<redacted>]
===================================================================================
STEP: Deleting objects not in source ...
Azure Data Factory (instance) loaded.
DataSets: 0 object(s) loaded.
IntegrationRuntimes: 0 object(s) loaded.
LinkedServices: 4 object(s) loaded.
Pipelines: 10 object(s) loaded.
DataFlows: 0 object(s) loaded.
Triggers: 6 object(s) loaded.
===================================================================================
STEP: Starting all triggers...
The number of triggers to start: 0
==============================================================================
   *****   Azure Data Factory files have been deployed successfully.   *****

Data Factory name:  <redacted>
Region (Location):  <redacted>
     Elapsed time:  0:01:02.939

==============================================================================

=========================================================================================================
    - How much helpful this extension Task for Azure Data Factory is?                                    
    - If you like it, if it saves your time or maybe something could be done better?                     
    - I would be really appreciate if you rate this tool and leave your honest comment.                  
(https://marketplace.visualstudio.com/items?itemName=SQLPlayer.DataFactoryTools&ssr=false#review-details)
                         THE COMMUNITY WOULD LOVE TO HEAR YOUR FEEDBACK !                                
                                            Me either :)                                        Thanks!  
=========================================================================================================

##[section]Finishing: Deploy Data Factory
clpalmer commented 2 years ago

@NowinskiK - And also, if it's of any use, here's the task config, though there's nothing special there

NowinskiK commented 2 years ago

Hi @clpalmer The message The number of triggers to start: in the log shows how many ACTIVE triggers are in the code to be started. If you see zero, it simply means you probably have runtimeState = Stopped or the config you have changing this property. Do you have any lines in config to change triggers?

clpalmer commented 2 years ago

Thanks @NowinskiK. Indeed was a misunderstanding on my part. It's not stopping whatever is running and then restarting what was stopped, it's stopping everything and then restarting what was in the deployed template. Closing this ticket, thanks for the input.