azurerm_data_factory_pipeline is not applying "escapeQuotesEscaping" property (in activities_json)

[miguelduarte18]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.5.2

AzureRM Provider Version

3.70.0

Affected Resource(s)/Data Source(s)

azurerm_data_factory_pipeline

Terraform Configuration Files

terraform {
  backend "local" {
    # Overridden by backend file dependent on environment being deployed to.
  }
}

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.70.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test_resource_group" {
    name     = "test_resource_group"
    location = "uksouth"
}

resource "azurerm_data_factory" "data_factory_test" {
  name                            = "data-factory-test"
  location                        = azurerm_resource_group.rg.location
  resource_group_name             = azurerm_resource_group.rg.name

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_data_factory_integration_runtime_azure" "test_runtime" {
  name            = "test_runtime"
  data_factory_id = azurerm_data_factory.partners_data.id
  location        = "your_location"
  description     = "Integrated runtime for Azure Data Factory pipelines"
}

resource "azurerm_data_factory_pipeline" "test_pipeline_quotes" {
  name            = "test_quotes_escaping_pipeline"
  data_factory_id = azurerm_data_factory.partners_data.id
  activities_json = templatefile("./activities_test_quotes.json", {
  })
}

activities_test_quotes.json:
[
    {
        "name": "CopyDataTestActivity",
        "type": "Copy",
        "policy": {
            "timeout": "0.12:00:00",
            "retry": 1,
            "retryIntervalInSeconds": 30,
            "secureOutput": false,
            "secureInput": false
        },
        "userProperties": [],
        "typeProperties": {
            "source": {
                "type": "JiraSource",
                "query": {
                    "value": "select 1 as testcol",
                    "type": "Expression"
                }
            },
            "sink": {
                "type": "SnowflakeSink",
                "importSettings": {
                    "type": "SnowflakeImportCopyCommand"
                }
            },
            "enableStaging": true,
            "stagingSettings": {
                "linkedServiceName": {
                    "referenceName": "linked_service_xxxx",
                    "type": "LinkedServiceReference"
                }
            },
            "escapeQuoteEscaping": true
        },
        "inputs": [
            {
                "referenceName": "source_xxxx",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "destination_xxxx",
                "type": "DatasetReference"
            }
        ]
    }
]

Debug Output/Panic Output

Nothing to add here.

Expected Behaviour

A new pipeline is created, including the property "escapeQuoteEscaping": true.

Actual Behaviour

The "escapeQuoteEscaping" property is always present in terraform plan but then it is not applied. More details on what happens, step by step:

1. Running terraform plan produces a nice looking plan:

   
   Terraform will perform the following actions:
   
   # module.jira-pipeline.azurerm_data_factory_pipeline.test_pipeline_quotes will be created
   + resource "azurerm_data_factory_pipeline" "test_pipeline_quotes" {
     + activities_json = jsonencode(
           [
             + {
                 + inputs         = [
                     + {
                         + referenceName = "source_xxxx"
                         + type          = "DatasetReference"
                       },
                   ]
                 + name           = "CopyDataTestActivity"
                 + outputs        = [
                     + {
                         + referenceName = "destination_xxxx"
                         + type          = "DatasetReference"
                       },
                   ]
                 + policy         = {
                     + retry                  = 1
                     + retryIntervalInSeconds = 30
                     + secureInput            = false
                     + secureOutput           = false
                     + timeout                = "0.12:00:00"
                   }
                 + type           = "Copy"
                 + typeProperties = {
                     + enableStaging       = true
                     + escapeQuoteEscaping = true
                     + sink                = {
                         + importSettings = {
                             + type = "SnowflakeImportCopyCommand"
                           }
                         + type           = "SnowflakeSink"
                       }
                     + source              = {
                         + query = {
                             + type  = "Expression"
                             + value = "select 1 as testcol"
                           }
                         + type  = "JiraSource"
                       }
                     + stagingSettings     = {
                         + linkedServiceName = {
                             + referenceName = "linked_service_xxxx"
                             + type          = "LinkedServiceReference"
                           }
                       }
                   }
                 + userProperties = []
               },
           ]
       )
     + data_factory_id = "**********"
     + id              = (known after apply)
     + name            = "test_quotes_escaping_pipeline"
   }

Plan: 1 to add, 0 to change, 0 to destroy.

2. Run `terraform apply` to deploy the pipeline, and inspected the pipeline manually (in Azure UI). The pipeline JSON does not have the "escapeQuotesEscaping" property. This is how the JSON on ADF looks like:

{ "name": "test_quotes_escaping_pipeline", "properties": { "activities": [ { "name": "CopyDataTestActivity", "type": "Copy", "dependsOn": [], "policy": { "timeout": "0.12:00:00", "retry": 1, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "userProperties": [], "typeProperties": { "source": { "type": "JiraSource", "query": { "value": "select 1 as testcol", "type": "Expression" } }, "sink": { "type": "SnowflakeSink", "importSettings": { "type": "SnowflakeImportCopyCommand" } }, "enableStaging": true, "stagingSettings": { "linkedServiceName": { "referenceName": "linked_service_xxxx", "type": "LinkedServiceReference" } } }, "inputs": [ { "referenceName": "destination_xxxx", "type": "DatasetReference" } ], "outputs": [ { "referenceName": "source_xxxx", "type": "DatasetReference" } ] } ], "annotations": [], "lastPublishTime": "2023-10-09T12:04:16Z" }, "type": "Microsoft.DataFactory/factories/pipelines" }

3. Tried to re-run `terraform plan`, and to my surprise, there was a detected change. This is how the plan looks.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place

Terraform will perform the following actions:

module.jira-pipeline.azurerm_data_factory_pipeline.test_pipeline_quotes will be updated in-place

~ resource "azurerm_data_factory_pipeline" "test_pipeline_quotes" { ~ activities_json = jsonencode( ~ [ ~ { name = "CopyDataTestActivity" ~ typeProperties = {

• escapeQuoteEscaping = true

  (4 unchanged attributes hidden)

              }
              # (5 unchanged attributes hidden)
          },
      ]
  )
  id              = "*****"
  name            = "test_quotes_escaping_pipeline"
  # (5 unchanged attributes hidden)

  }

Plan: 0 to add, 1 to change, 0 to destroy.

5. Ran `terraform apply`, but sadly got the same results - the property was not applied.
6. Applied the property manually via the UI.
7. Ran `terraform plan`, and had the exact same plan as before.
8. Ran `terraform apply` and the property that I manually added was removed.

### Steps to Reproduce
1. Run `terraform apply` for all resources except for the last one (pipeline `test_pipeline_quotes`);
2. Create 2 datasets in Azure Data Factory UI: `source_xxxx` and `destination_xxxx`. I used jira for the source and Snowflake for the destination but in practice it should not matter much. If you use different ones, make sure to update the type in the JSON file (as of now it is `JiraSource` and `SnowflakeSink`);
3. Remaining instructions to help reproduce are stated in the  "Actual Behaviour".

### Important Factoids

_No response_

### References

_No response_

[Azilyss]

I have the same issue, did you find a workaround ?