search

Azure-Player / azure.datafactory.devops

Azure DevOps Deployment Tasks for Azure Data Factory objects
MIT License
39 stars 21 forks source link

readme

Azure DevOps Tasks (#adftools)

This extension to Azure DevOps has three tasks and only one goal: deploy Azure Data Factory (v2) seamlessly and reliable at minimum efforts. As opposed to ARM template publishing from 'adf_publish' branch, this task publishes ADF directly from JSON files, who represent all ADF artefacts.
The task based on the PowerShell module azure.datafactory.tools available through PSGallery.
Fully written in PowerShell, compatible with Windows PowerShell 5.1, PowerShell Core 6.0 and above.

Three tasks

Bear in mind that these tasks works only for Azure Data Factory v2.

How to add task

For classic pipelines, you will find the Tasks available under the Deploy tab, or search for adftools: Adding Task

You can also use the tasks in a pipeline configured with a yml file, the three tasks are called SQLPlayer.DataFactoryTools.BuildADF.BuildADFTask, SQLPlayer.DataFactoryTools.PublishADF.PublishADFTask and SQLPlayer.DataFactoryTools.TestADFLS.TestAdfLinkedServiceTask. You can also use the DevOps UI to configure these tasks and then view the generated yml, to ensure the yml is correct. One example, in which we use the 'Build ADF' task:

- job: 'build_adf'
  displayName: 'Build Azure Data Factory'
  pool:
    vmImage: 'windows-latest'
  steps:
    - task: SQLPlayer.DataFactoryTools.BuildADF.BuildADFTask@1
      displayName: 'Validate ADF files'
      inputs:
        DataFactoryCodePath: '$(System.DefaultWorkingDirectory)/data-factory'
        Action: Build

Publish Azure Data Factory

Use this task to deploy a folder of ADF objects from your repo to target Azure Data Factory instance.
For YAML pipelines, use task PublishADFTask@1.

Key capabilities

For more details, please go to documentation of azure.datafactory.tools.

Parameters

Parameter label Parameter name (for YAML task) Description
Azure Subscription azureSubscription Azure subscription to target for deployment
Resource Group Name ResourceGroupName Provide the name of the Resource Group
Target Azure Data Factory Name DataFactoryName Provide the name of the target Azure Data Factory
Azure Data Factory Path DataFactoryCodePath Path from the repo root to the ADF folder which should contains sub-folders like 'pipeline', 'dataset', etc.
Target Region Location Azure Region of target Data Factory. Required, but used only when creating a new instance of ADF
Environment Config Type StageType Specifies how you would provide set of parameters for Stage.
Environment (stage) StageCode Allows pointing configuration with values for all properties who need to be replaced.
If parameter is specified, a CSV file named './deployment/config-{stage}.csv' must exist in repo.
Environment (stage) Config File Path StageConfigFile Allows pointing configuration with values for all properties who need to be replaced.
If specified, CSV config file name must ends with '.csv'
Delete objects not in source DeleteNotInSource Indicates whether the deployment process should remove objects not existing in the source (code)
Stop/Start triggers StopStartTriggers Indicates whether to stop the triggers before beginning deployment and start them afterwards
Create new ADF instance CreateNewInstance Indicates whether to create a new ADF if target instance doesn't exist yet.
Filtering Type FilteringType Type of filtering ADF objects: File Path or Inline Text Field
Include/Exclude Filtering Text FilterText Multi-line or comma-separated list of objects to be included or excluded in the deployment.
For example, see below.
Include/Exclude Filtering File Path FilterTextFile Multi-line or comma-separated list of objects to be included/excluded in/from the deployment.
For example, see below.
Do not Stop/Start excluded triggers DoNotStopStartExcludedTriggers Specifies whether excluded triggers will be stopped before deployment.
Do not delete excluded objects DoNotDeleteExcludedObjects Specifies whether excluded objects can be removed. Applies when DeleteNotInSource is set to True only.
Ignore lack of referenced object IgnoreLackOfReferencedObject Usually, the deployment will fail if a resource is referenced that's not present in a JSON file. Set to True to warn instead of failing.

Environment (stage)

Optional parameter. When defined, process will replace all properties defined in (csv) configuration file. The parameter can be either full path to csv file (must ends with .csv) or just stage name. When you provide parameter value 'UAT' the process will try open config file located in .\deployment\config-UAT.csv

More details: Step: Replacing all properties environment-related

Selective deployment

The task allows you to deploy subset of ADF's objects.
You can select objects specifying them by object's type, name or folder which belongs to, using include or exclude option.
All 3 parts (Type, Name, Folder) can be wildcarded, so all such variants are possible:

You can specify them by exact name or wildcard.
Example: 

+pipeline.PL_Copy*  
+dataset.ds_srcCopy  
dataset.*  
-pipeline.PL_DoNotPublish*  
-integrationruntime.*
-*.*@testFolder

To simplify user experience – only one field is exposed in order to define include/exclude rules. Therefore, an extra character should be provided before the name/pattern:

If char (+/-) is not provided – an inclusion rule would be applied.

Read more: Selective deployment, triggers and logic

Screenshot of Publish Task

Task

Build/Test Azure Data Factory code

Another very helpful task is Build Azure Data Factory. The task has two actions to be chosen:

Build only

Use this action to validate the code of your Azure Data Factory before you publish it onto target ADF service. The function validates files of ADF in a given location, returning warnings or errors. This validation functionality is implemented in this task, so it is not the same implementation as behind the 'Validate all' button in the ADF UI. The following validation will be performed:

The task sets these 2 pipeline output variables:

You can use them in any subsequent tasks as any other DevOps variables: $(AdfBuildTaskErrors)

Validate & Export ARM Template

This action uses ADFUtilities NPM package provided by Microsoft. It does exactly the same actions as you can do with ADF UI by clicking Validate all and then Export ARM Template. Therefore, this is a different implementation as the other action described above. So finally, you can automate this step and fully automate ADF deployment, even if you prefer to use Microsoft's approach with ARM Template.

Parameters:

Screenshot of Build Task

Task

Test connection of ADF Linked Service

After deployment within automated CI/CD process, we would like to test foundamental Linked Services whom are used for dependant objects in ADF. In order to ensure the entire workload works, we must make sure that all Linked Services are configured correctly and have access to pointing resources.
The purpose of this task is to ensure such checking. It works exactly the same as hitting button Check connection in ADF Linked Service.

Be aware that the task uses undocumented API function.

Credits: This task arose thanks to Simon D'Morias based on his blog post.

Screenshot of Test Connection Task

Task

Related modules

These tasks include the following modules:

History

Version 2

Version 1