Terraform deploys Data Factory Linked Service in Live Mode Only

akinsriv commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.1.7

AzureRM Provider Version

3.0.2

Affected Resource(s)/Data Source(s)

azurerm_data_factory_linked_service_key_vault

Terraform Configuration Files

resource "azurerm_data_factory" "this" {
  name                = module.wf_datafactory_name.name
  resource_group_name = data.azurerm_resource_group.this.name
  location            = module.wf_datafactory_name.location

  dynamic "github_configuration" {
    for_each = try(var.github_configuration, null) != null ? [var.github_configuration] : []

    content {
      account_name    = github_configuration.value.account_name
      branch_name     = github_configuration.value.branch_name
      git_url         = github_configuration.value.git_url
      repository_name = github_configuration.value.repository_name
      root_folder     = github_configuration.value.root_folder
    }
  }
  dynamic "global_parameter" {
    for_each = coalesce(var.global_parameter, [])

    content {
      name  = global_parameter.value.name
      type  = global_parameter.value.type
      value = global_parameter.value.value
    }
  }
  identity {
    type         = "UserAssigned"
    identity_ids = local.identity_ids
  }

  managed_virtual_network_enabled  = try(var.managed_virtual_network_enabled, null)
  public_network_enabled           = true
  customer_managed_key_id          = try(azurerm_key_vault_key.cmk.id)
  tags                             = local.tags

  depends_on = [module.wf_uai_datafactory, azurerm_key_vault_key.cmk]
}
resource "azurerm_data_factory_linked_service_key_vault" "linked_service_key_vault" {
  name                     = module.wf_datafactory_ls_kv_name.name
  data_factory_id          = azurerm_data_factory.this.id
  description              = var.description
  integration_runtime_name = var.integration_runtime_name
  annotations              = var.annotations
  parameters               = var.parameters
  additional_properties    = var.additional_properties
  key_vault_id             = data.azurerm_key_vault.this.id
}

Debug Output/Panic Output

NA

Expected Behaviour

The linked service must be created in GIT Mode.

Actual Behaviour

The linked service must be created in Live Mode.

Steps to Reproduce

No response

Important Factoids

No response

References

No response

mkprizzle commented 2 years ago

In my testing, I am not even able to get a KV linked resource to show up in data factory when deployed with Terraform using the documentation example code. Is the resource itself working correctly?

Running the az cli command it does indeed show up, but I'm not aware of how to tell which mode it is (also, not showing up as a linked service in adf.azure.com)

az datafactory linked-service list --factory-name "testdf1234754324" --resource-group "example-resources"

[  
  {
    "etag": "<eTag>",
    "id": "/subscriptions/<sub-ID>/resourceGroups/example-resources/providers/Microsoft.DataFactory/factories/testdf1234754324/linkedservices/linked_kv",
    "name": "linked_kv",
    "properties": {
      "additionalProperties": null,
      "annotations": null,
      "baseUrl": "https://dfkeyvault1269492.vault.azure.net/",
      "connectVia": null,
      "credential": null,
      "description": "",
      "parameters": null,
      "type": "AzureKeyVault"
    },
    "resourceGroup": "example-resources",
    "type": "Microsoft.DataFactory/factories/linkedservices"
  }
]

akinsriv commented 2 years ago

In my testing, I am not even able to get a KV linked resource to show up in data factory when deployed with Terraform using the documentation example code. Is the resource itself working correctly?

Running the az cli command it does indeed show up, but I'm not aware of how to tell which mode it is (also, not showing up as a linked service in adf.azure.com)

az datafactory linked-service list --factory-name "testdf1234754324" --resource-group "example-resources"
[  
  {
    "etag": "<eTag>",
    "id": "/subscriptions/<sub-ID>/resourceGroups/example-resources/providers/Microsoft.DataFactory/factories/testdf1234754324/linkedservices/linked_kv",
    "name": "linked_kv",
    "properties": {
      "additionalProperties": null,
      "annotations": null,
      "baseUrl": "https://dfkeyvault1269492.vault.azure.net/",
      "connectVia": null,
      "credential": null,
      "description": "",
      "parameters": null,
      "type": "AzureKeyVault"
    },
    "resourceGroup": "example-resources",
    "type": "Microsoft.DataFactory/factories/linkedservices"
  }
]
Thanks @mkprizzle. However, the above terraform code does create a Key Vault Linked service for me. The given code has mentions of some name coded modules and data reference to Key Vault from where it fetched ID. Also, once successfully executed it does take a minute or two for resource to show up in Azure Data Factory Studio.

For the latter question, if you navigate to Azure portal you can look up if your resource is created in "Live Mode" or "Git Mode", such as below, if haven't already tried:

ms-henglu commented 2 years ago

Hi @akinsriv ,

Thanks for taking time to report this issue.

I'm not sure whether I understand this correctly, but from the image you attached, it says Switch to live mode which means currently it's in git mode?

akinsriv commented 2 years ago

Hi @akinsriv ,

Thanks for taking time to report this issue.

I'm not sure whether I understand this correctly, but from the image you attached, it says Switch to live mode which means currently it's in git mode?

I have posted that image, to show mkprizzle where to switch between the modes. Basically, I only see the Data Linked Service deployed when I go to Live Mode, i.e, Data Mode, however I want it to be visible under Git Mode. By portal I can do it but not via Terraform.

ms-henglu commented 2 years ago

Authoring directly with the Data Factory service is disabled in the Azure Data Factory UX when a Git repository is configured. Changes made via PowerShell or an SDK are published directly to the Data Factory service, and are not entered into Git.

From the official document, it's not impossible for terraform to support this feature.

Ref: https://docs.microsoft.com/en-us/azure/data-factory/source-control

dan-corneanu commented 8 months ago

When in GIT mode, all linked services definitions are read from the template files in the GIT branch that was configured for this factory. Thus, terraform definitions will only be visible in live mode.

dankellett commented 3 months ago

Authoring directly with the Data Factory service is disabled in the Azure Data Factory UX when a Git repository is configured. Changes made via PowerShell or an SDK are published directly to the Data Factory service, and are not entered into Git.

From the official document, it's not impossible for terraform to support this feature.

Ref: https://docs.microsoft.com/en-us/azure/data-factory/source-control

It looks like this is still true, but can someone point out where in the linked documentation this is stated?

hashicorp / terraform-provider-azurerm