Invalid 'dependsOn' attribute should fail BuildADFTask

[PHPirates]

We are using the BuildADFTask to validate the ADF files in our CI pipeline, but if there is a pipeline in which an activity has a dependsOn attribute on an activity that does not actually exist, the validate step succeeds even though when run the pipeline will fail. Is this a bug in the Data Factory itself, or can it be fixed in this task?

Example pipeline:

{
    "name": "pipeline1",
    "properties": {
        "activities": [
            {
                "name": "Wait1",
                "type": "Wait",
                "dependsOn": [],
                "userProperties": [],
                "typeProperties": {
                    "waitTimeInSeconds": 1
                }
            },
            {
                "name": "Wait2",
                "type": "Wait",
                "dependsOn": [
            {
                "activity": "does-not-exist",
                "dependencyConditions": [
                    "Succeeded"
                ]
            }
        ],
                "userProperties": [],
                "typeProperties": {
                    "waitTimeInSeconds": 1
                }
            }
        ],
        "annotations": []
    }
}

It validates, but produces this error when run:

ErrorCode=InvalidTemplate, ErrorMessage=The template validation failed: 'The 'runAfter' property of template action 'bad_lossScope' contains non-existent action: 'employmentScope'

Our yml:

- job: 'adf'
  displayName: 'Data Factory'
  pool:
    vmImage: 'windows-latest'
  steps:
    - task: SQLPlayer.DataFactoryTools.BuildADF.BuildADFTask@1
      displayName: 'Validate ADF files'
      inputs:
        DataFactoryCodePath: '$(System.DefaultWorkingDirectory)/data-factory'

It isn't clear to me whether this task is using the same functionality as the 'validate all' button in the UI, I'm confused by the readme: it has a 'Build only' section, and then a 'Validate & Export ARM Template' section which mentions the azure-data-factory-utilities npm package, but also a 'Build only' action of which I don't know whether it is the same as the 'Build only' of the previous section. I looked in the source and it does seem like some checks are implemented in this repo.

[NowinskiK]

There are two actions you can choose from in this task:

• "Build": "Build only (simple validate)",
• "Export": "Validate & Export ARM Template"

The build (default) is developed internally and has nothing to do with Microsoft's "Validate All" button known from UI. This is your case as you did not specify Action parameter. Currently, the task does NOT support the validation of activities dependencies. It does it for ADF objects dependencies (like pipeline, dataset, linked service, etc).

Export executes export command from Microsoft's (npm) library, which includes validation operation as well. This action is a counterpart of "Validate All" button known from ADF UI.

In the readme file both sections are mentioned: https://github.com/SQLPlayer/azure.datafactory.devops#buildtest-azure-data-factory-code 1) Build only 2) Validate & Export ARM Template

[PHPirates]

The build (default) is developed internally and has nothing to do with Microsoft's "Validate All" button known from UI.

That was not clear from the readme, but I understand. The functionality seems quite similar to me.

Currently, the task does NOT support the validation of activities dependencies.

That's indeed what I found out, but I can understand if you don't want to implement it, because the Data Factory 'Validate All' button hasn't implemented this either. We'll just implement our own check for that, then.

In the readme file both sections are mentioned:

1. Build only
2. Validate & Export ARM Template

Of course I saw this, but now from your comment I can guess that the 'build only (validate)' of item 1 refers to the internally developed 'build only' of the previous section, and it is not the same as the 'validate' from item 2.

Basically it is not clear to me from the readme what yml code I should use, the way I now do it is first use a DevOps pipeline to add these tasks using the UI, and then view the yml, which also works. [Edit] Using the DevOps UI I now see that it's the BuildADFTask which has the 'action' parameter, so that clarifies. It's called Action: Export though, which does not match with the readme, but now I see that this is what you meant in your comment.

Thanks for the support!

[NowinskiK]

Thanks for your comments, I really appreciate it. You're probably right - the documentation might be sorted out and enhanced in some section (especially this one), which I will do in my spare time. If you're happy to share your code with me and adopt it in this module, you're welcome. Let me know if you prefer to just send me the code or make the changes and create PR on your own.

[PHPirates]

Thanks, I can also help with the documentation, I can create a PR later today. I'm not very fluent in PowerShell, so I will have a look next week or so if I can figure it out the code, and if so I'll create a PR for sure!