Rewrite ADF Pipelines to Synapse Data Pipelines

[devlace]

Description

Using the Parking Sensor ADF Data Pipelines as a base, update this to work for Synapse pipelines. Changes include, but not limited to:

• Removing the Ls_AzureDatabricks_01 Linked Service
• Updating the Ls_AzureSQLDW_01 LinkedService to use Managed Identity
• Updating the pipeline to call Synapse Spark Notebooks instead of Databricks notebooks

Original Parking Sensor ADF artifact definitions: https://github.com/Azure-Samples/modern-data-warehouse-dataops/tree/main/e2e_samples/parking_sensors/adf

Dependencies

• 394

DoD

• E2E working Synapse Data Pipeline.
• Demonstrate that Synapse SQL Dedicated Pool (SQL DW) contains output dataset after pipeline run.

[icchan]

In the most recent version of the engagements' synapse archi, I rewrote the linked service to be a key vault reference. It makes the CD slightly easier since we don't need to override this linked service (only the KV one)

Do we want to do that for this sample too?

e.g.

{
    "name": "pipelineADLS",
    "type": "Microsoft.Synapse/workspaces/linkedservices",
    "properties": {
        "type": "AzureBlobFS",
        "annotations": [],
        "typeProperties": {
            "url": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "keyvault",
                    "type": "LinkedServiceReference"
                },
                "secretName": "Synapse-PipelineAdls-Url"
            }
        },
        "connectVia": {
            "referenceName": "AutoResolveIntegrationRuntime",
            "type": "IntegrationRuntimeReference"
        }
    }
}

[devlace]

In the most recent version of the engagements' synapse archi, I rewrote the linked service to be a key vault reference. It makes the CD slightly easier since we don't need to override this linked service (only the KV one)

Do we want to do that for this sample too?

e.g.

{
  "name": "pipelineADLS",
  "type": "Microsoft.Synapse/workspaces/linkedservices",
  "properties": {
      "type": "AzureBlobFS",
      "annotations": [],
      "typeProperties": {
          "url": {
              "type": "AzureKeyVaultSecret",
              "store": {
                  "referenceName": "keyvault",
                  "type": "LinkedServiceReference"
              },
              "secretName": "Synapse-PipelineAdls-Url"
          }
      },
      "connectVia": {
          "referenceName": "AutoResolveIntegrationRuntime",
          "type": "IntegrationRuntimeReference"
      }
  }
}

The existing Parking Sensor sample should already be using KV as the go-to secret store where possible. I will just expand the scope of this story to update the ADF Data Pipelines in general from the Parking Sensor solution including the LinkedServices to work for the Synapse version of the solution.