DeleteNotInSource might enter in infinite loop

[henriquebrisola]

When using DeleteNotInSource as True, but the deploy cannot delete an object, it might enter in an infinite loop trying to delete over and over again.

In my case what happened was that a linked service was being referenced by the SSIS Integration Runtime, however the options were as follows:

$opt.DeleteNotInSource = $true
$opt.Excludes.Add("integrationruntime.*", "")

In other words, it tried to delete a linked services without removing its reference from the SSIS-IR, which was set to not touch it. You might be able to reproduce with other objects by trying to delete something that is reference by something else that is excluded on the deploy options.

I believe a failure message should be provided and there shouldn't be any more than 5 tries for each object.

Because it didn't fail I got a timeout from the azure devops agent and all triggers stayed disabled.

[NowinskiK]

Thanks for raising this, @henriquebrisola

[NowinskiK]

BTW. I see that you have over 3000 objects in your ADF. How long does it take to deploy all of them?

[henriquebrisola]

BTW. I see that you have over 3000 objects in your ADF. How long does it take to deploy all of them?

It tried full deploy after the incremental fails, it takes around 40 minutes altogether.

[NowinskiK]

Do you mean new "Incremental deployment" feature by "Incremental"? If so, why it failed?

[henriquebrisola]

Oh no, haven't looked into yours incremental feature, I actually implemented one using -FilterFilePath Basically I use git diff -M --name-only HEAD HEAD~1 to give me all code changes and list them in a file saying that they should be considered for deployment. The file gets like below, first removing everything from deployment, then adding using the gif diff command:

-dataflow.*
-dataset.*
-linkedService.*
-pipeline.*
-template.*
-trigger.*
+pipeline.pipeline_name
[...]

But I will take a look into the new feature.

[NowinskiK]

Thanks for sharing your approach. Pretty interesting, however, not quite right. For example: how do you know which 2 commits you need to compare? The latest one and ...? Try new feature: Incremental Deployment. Anyway... I'm not sure whether this case is a bug. When you decide to include/exclude objects from deployment - you must be sure you're defining this correctly. In this case integrationruntime should be removed first (means: included), or the pipeline shouldn't be deleted in the first place. Can you elaborate what happened in dev environment?