PrePostDeploymentScript.Ver2 Fails on Start Triggers - "Getting pipelines"

[mariobgallardo]

I'm using the version 2 from PrePostDeploymentScript on my CI/CD ADF, but for some reason when it reaches the step "getting pipelines" it fails, as shown bellow:

Getting triggers Getting pipelines WARNING: The output of cmdlet Get-AzDataFactoryV2Pipeline may compromise security by showing the following secrets: Activities.Url, Activities.AdditionalProperties. Learn more at https://go.microsoft.com/fwlink/?linkid=2258844

[error] Index operation failed; the array index evaluated to null. from Line: 82

ForEach-Object: /home/vsts/work/1/s/powershell_script/PrePostDeploymentScript.ps1:82 Line | 82 | … elineNameResourceDict $pipelineNameResourceDict } | ForEach-Object { | ~~~~ | Index operation failed; the array index evaluated to null.

[error]PowerShell exited with code '1'.

It looks like the parameter $pipelineNameResourceDict is empty. I'm confused because I'm using the same script (changing the specific parameters) on another ADF environment and it works perfectly. Besides that, the other ADF environment is like a copy from this, so they are pretty much equal.

[Liszet]

I am facing the same issue in our deployment pipelines with both v2 and v1 actually. This has worked ok before on this pipeline.

[Liszet]

@mariobgallardo i have found the fix! The problem is when one has an execute pipeline in the ADF, that references a pipeline that has been deleted, so the activity is null, and this is what causes the problem. The fix is to go through the execute pipelines activities in your ADF and remove the ones that do not have an invoked pipeline chosen anymore. These issues are not caught by the 'Validate' button, so I think this is where the problem should first be discovered. I have therefore created a support ticket with ADF team so that this is caught by the Validate action.

To further help in identifying these pipelines I have temporarily added more debugging to the Push-PipelinesToList function

# Function to sort pipelines based on their dependencies
function Push-PipelinesToList {
    param(
        [Microsoft.Azure.Commands.DataFactoryV2.Models.PSPipeline]$pipeline,
        [Hashtable] $pipelineNameResourceDict,
        [Hashtable] $visited,
        [System.Collections.Stack] $sortedList
    )
    if ($visited[$pipeline.Name] -eq $true) {
        return;
    }
    $visited[$pipeline.Name] = $true
    $pipeline.Activities | ForEach-Object { 
        $activityName = $_.Name
        Get-PipelineDependency -activity $_ -pipelineNameResourceDict $pipelineNameResourceDict 
    } | ForEach-Object {
        if ($_ -eq $null) {
            Write-Warning "Pipeline dependency is null for pipeline $($pipeline.Name) and activity $activityName. The activity $activityName is missing an invoked pipeline reference."
        } else {
            Push-PipelinesToList -pipeline $pipelineNameResourceDict[$_] -pipelineNameResourceDict $pipelineNameResourceDict -visited $visited -sortedList $sortedList
        }
    }
    $sortedList.Push($pipeline)
}

[sandman153]

Hi @Liszet

Thanks for your help. I will need to check my code next week and post back. I believe the error maybe caused by the same problem.

Kind regards, Sarath

[Liszet]

@sandman153 Would be nice to know if you have gotten further with your debugging.

I have been in touch with the Azure Data Factory team about catching this problem in the Validate step, and they have registered my request. If you are facing the same issue, it would be great if you can also send a support request to ADF team. If more people express interest for this fix, then the team can prioritize this.

[sandman153]

Hi @Liszet

Apologies for the late response. Yes I believe I am hitting the same problem as you.

I put in a lot of workarounds using PowerShell (I ended up adding a filter to my code to filter out the ADF pipeline causing the issue).

[Liszet]

@sandman153 The ADF product team just fixed the validation in the Data Factory, so now the execute pipelines activities that have no invoked pipeline should show up in the validate step during development. You can see if the pipeline you had to filter out is now showing up with issues in the validation.

[Liszet]

@mariobgallardo The ADF product team just fixed the validation in the Data Factory, so now the execute pipelines activities that have no invoked pipeline should show up in the validate step during development. Take a look and see if you now get validation errors in the ADF project where you got the error in the deployment script

[mariobgallardo]

Hello @Liszet. Sorry for the late response. I'm just about to finish a project here and I'll return to this subject next week. As soon as I run it I will let you know. Thank you very much for your help!!

[mariobgallardo]

Hello @lizset. I tried the new Prepostdeploymentscript from MS and it worked fine on this ADF environment, which was not working. But now the other ADF environment fails on Start ADF when it reaches the "function Get-SortedLinkedService". Does it happens to you too?

[Liszet]

Sorry, I am not facing this issue, so cannot help there. My deployment is working ok now.

On Fri, 23 Aug 2024 at 14:58, Mario Gallardo @.***> wrote:

Hello @Lizset https://github.com/Lizset. I tried the new Prepostdeploymentscript from MS and it worked fine on this ADF environment, which was not working. But now the other ADF environment fails on Start ADF when it reaches the "function Get-SortedLinkedService". Does it happens to you too?

— Reply to this email directly, view it on GitHub https://github.com/Azure/Azure-DataFactory/issues/679#issuecomment-2307041885, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKYUUCNC5UY4MK57HFWSWKTZS4WWJAVCNFSM6AAAAABKCJIZXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBXGA2DCOBYGU . You are receiving this because you were mentioned.Message ID: @.***>

[mariobgallardo]

Issue Fixed for "Getting pipelines". Now there's a problem on "Getting linked services"