Azure-Player / azure.datafactory.tools

Tools for deploying Data Factory (v2) in Microsoft Azure
https://azureplayer.net/adftools
MIT License
207 stars 69 forks source link

Incremental re-deployment of deleted objects #355

Closed AttilaLorincz closed 10 months ago

AttilaLorincz commented 1 year ago

Describe the bug If an object is deleted, it is not recreated on further incremental deployments, if it did not change.

To Reproduce

  1. Do a deployment from a codebase where object O exists created from O.json
  2. Delete O.json from the codebase
  3. Deploy again (with options IncrementalDeployment: true, DeleteNotInSource: true) Note: This will delete the object O. The entry for object O is not removed from the adftools_deployment_state global parameter.
  4. Re-add the same json file for object O to the codebase
  5. Deploy again (with options IncrementalDeployment: true, DeleteNotInSource: true) Note: the object is not created because the hash calculated from the json file matches the value saved in adftools_deployment_state

Expected behaviour An object that does not exist in live mode should be created by the incremental deployment.

Version of PowerShell module of: Version 1.6.003

Provide all files to reproduce the issue Relevant parts of log (Options) ... Includes : {} Excludes : {managedPrivateEndpoint., integrationruntime., managedVirtualNetwork.} DeleteNotInSource : True StopStartTriggers : True CreateNewInstance : False DeployGlobalParams : True FailsWhenConfigItemNotFound : True FailsWhenPathNotFound : True IgnoreLackOfReferencedObject : False DoNotStopStartExcludedTriggers : False DoNotDeleteExcludedObjects : True IncrementalDeployment : True TriggerStopMethod : DeployableOnly TriggerStartMethod : KeepPreviousState

... The non-deployed object will show up in the list of unchanged objects explaining why it's skipped STEP: Determining the objects to be deployed... VERBOSE: Incremental Deployment = True VERBOSE: The following objects will not be deployed as they have no changes since last deployment:

NowinskiK commented 12 months ago

Thank you @AttilaLorincz for raising this bug. It will be fixed as soon as I can.

MartaUch commented 10 months ago

Hello @NowinskiK, As this bug influence on our decision to use "incremental deployment" I would like to kindly ask you if you could provide time period when we could expect resolution of the bug (in three weeks, three months) :) Of course no pressure, but we are just wondering when we could start working on the extension of the app usage with this feature. Kind Regards Marta Uchimiak

NowinskiK commented 10 months ago

Weeks, probably this month.

MartaUch commented 10 months ago

Alright, perfect :) Thank you!

MartaUch commented 10 months ago

Hi :) I have another question. The solution will depend probably on the usage of DeleteNotInSource feature, correct? It means that if we are not using this feature and remove for now objects using customized power shell script after deployment, then we would have to adjust state of adftools_deployment_state parameter in this script by ourselves. Maybe you don't have solution yet in your mind, but I'm just wondering :)

NowinskiK commented 10 months ago

Marta, yes, you're 100% correct. You can start using Incremental feature even now and all works fine. This case is an edge-case scenario WHEN ALL the following points must be met: 1) Deploy with Incremental Mode + DeleteNotInSource 2) Delete object A 3) Recreate the same object (A) with exactly the same content. Only in this scenario object A will not be deployed. All would be fine in any of these: 3a) Recreate/create different object (B) 3b) Recreate object A with modification (even tiny)

PS. The solution is in my head for many weeks already (even two versions). If you want contribute in the module - you're welcome.

MartaUch commented 10 months ago

Alright, Thank you for confirming that. In such a case I will start to improve our Power Shell script to handle problem which is described here :)

Happy that you already have some solutions in mind and I’m not surprised you do :D If I have any idea worth of sharing I will contact you, thank you!