Support for passwords in Azure Key Vault for azurerm_data_factory_linked_service_sftp

[markhahnhtc]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

New or Affected Resource(s)

• azurerm_data_factory_linked_service_sftp

Potential Terraform Configuration

resource "azurerm_data_factory_linked_service_sftp" "ingestion_sftp_pyracloud" {
  name                          = "example-sftp-linked-service"
  resource_group_name           = azurerm_resource_group.example_resource_group.name
  data_factory_id               = azurerm_data_factory.example_adf.id

  authentication_type           = "Basic"
  host                          = "sftp.example.com"
  port                          = 22
  username                      = "example-sftp-account"
  password_key_vault            = "key-vault-linked-service"
  password_secret_name          = "example-sftp-secret-name"
  password_secret_version       = "latest" # or ("987226bf52054bf8b54207036e6f5210", a secret version id)
}

The password_key_vault, password_secret_name and password_secret_version are proposed. There may be better names or a more suitably complete solution.

References

See the information about using an Azure Key Vault here:
https://docs.microsoft.com/en-us/azure/data-factory/connector-sftp?tabs=data-factory#use-basic-authentication

[ms-henglu]

Hi @markhahnhtc , thank you for taking time to report this issue.

I have a workaround for this, how about using https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/data_factory_linked_custom_service? This is a generic resource that supports all different Linked Service Types.

[robertbrandso]

For others who experience the same limitation, here is an example using the workaround as mentioned by @ms-henglu:

# Create Data factory
resource "azurerm_data_factory" "main" {
  name                = "adf-example-lab"
  location            = var.location
  resource_group_name = azurerm_resource_group.main.name

  identity {
    type = "SystemAssigned"
  }
}

# Create linked services
## Key Vault
resource "azurerm_data_factory_linked_service_key_vault" "main" {
  name            = regex("[^/]+$", var.data_factory_linked_service_keyvault_id) # Using regex to extract Key Vault name. The name is at the end of the string after the slash ("/")
  data_factory_id = azurerm_data_factory.main.id
  description     = "Managed by Terraform"
  key_vault_id    = var.data_factory_linked_service_keyvault_id
}

## Give Data factory access to Key Vault secrets
resource "azurerm_key_vault_access_policy" "data_factory" {
  key_vault_id = var.data_factory_linked_service_keyvault_id
  object_id    = azurerm_data_factory.main.identity[0].principal_id
  tenant_id    = data.azurerm_client_config.current.tenant_id

  secret_permissions = ["Get"]
}

## Create SFTP linked service
resource "azurerm_data_factory_linked_custom_service" "sftp_example" {
  name            = "Example"
  data_factory_id = azurerm_data_factory.main.id
  description     = "Managed by Terraform"

  type                 = "Sftp"
  type_properties_json = <<JSON
{
  "host": "${var.data_factory_linked_service_example_sftp.host}",
  "port": ${var.data_factory_linked_service_example_sftp.port},
  "skipHostKeyValidation": true,
  "authenticationType": "Basic",
  "userName": "${var.data_factory_linked_service_example_sftp.username}",
  "password": {
      "type": "AzureKeyVaultSecret",
      "store": {
          "referenceName": "${azurerm_data_factory_linked_service_key_vault.main.name}",
          "type": "LinkedServiceReference"
      },
      "secretName": "${var.data_factory_linked_service_example_sftp.password_secret_name}"
  }
}
JSON

  depends_on = [azurerm_key_vault_access_policy.data_factory]
}

[antgustech]

For others who experience the same limitation, here is an example using the workaround as mentioned by @ms-henglu:

# Create Data factory
resource "azurerm_data_factory" "main" {
  name                = "adf-example-lab"
  location            = var.location
  resource_group_name = azurerm_resource_group.main.name

  identity {
    type = "SystemAssigned"
  }
}

# Create linked services
## Key Vault
resource "azurerm_data_factory_linked_service_key_vault" "main" {
  name            = regex("[^/]+$", var.data_factory_linked_service_keyvault_id) # Using regex to extract Key Vault name. The name is at the end of the string after the slash ("/")
  data_factory_id = azurerm_data_factory.main.id
  description     = "Managed by Terraform"
  key_vault_id    = var.data_factory_linked_service_keyvault_id
}

## Give Data factory access to Key Vault secrets
resource "azurerm_key_vault_access_policy" "data_factory" {
  key_vault_id = var.data_factory_linked_service_keyvault_id
  object_id    = azurerm_data_factory.main.identity[0].principal_id
  tenant_id    = data.azurerm_client_config.current.tenant_id

  secret_permissions = ["Get"]
}

## Create SFTP linked service
resource "azurerm_data_factory_linked_custom_service" "sftp_example" {
  name            = "Example"
  data_factory_id = azurerm_data_factory.main.id
  description     = "Managed by Terraform"

  type                 = "Sftp"
  type_properties_json = <<JSON
{
  "host": "${var.data_factory_linked_service_example_sftp.host}",
  "port": ${var.data_factory_linked_service_example_sftp.port},
  "skipHostKeyValidation": true,
  "authenticationType": "Basic",
  "userName": "${var.data_factory_linked_service_example_sftp.username}",
  "password": {
      "type": "AzureKeyVaultSecret",
      "store": {
          "referenceName": "${azurerm_data_factory_linked_service_key_vault.main.name}",
          "type": "LinkedServiceReference"
      },
      "secretName": "${var.data_factory_linked_service_example_sftp.password_secret_name}"
  }
}
JSON

  depends_on = [azurerm_key_vault_access_policy.data_factory]
}

Thank you, this worked perfectly. However, I also need to be using ssh keyes in another linked service, "authenticationType": "SshPublicKey"

Do you know what the rest of the properties are called? I cannot find them anywhere!

[robertbrandso]

@antgustech, it's been a while since I worked on this, but the way I found the properties to use in the example I posted was to create the linked service manually in the Data Factory portal, looked at the JSON after it was created (hover over the linked service -> click the {} icon) and adjusted it as needed to work with the Terraform configuration.

[antgustech]

@antgustech, it's been a while since I worked on this, but the way I found the properties to use in the example I posted was to create the linked service manually in the Data Factory portal, looked at the JSON after it was created (hover over the linked service -> click the {} icon) and adjusted it as needed to work with the Terraform configuration.

Ah yes, did that but that part is encrypted nowadays.

Edit:

I believe I figured in out, it should look like this:

resource "azurerm_data_factory_linked_custom_service" "this" {
  name                 = var.name
  data_factory_id      = var.data_factory_id
  type                 = "Sftp"
  type_properties_json = <<JSON
{
  "host": "${var.host}",
  "port": 22,
  "skipHostKeyValidation": false,
  "host_key_fingerprint": "${var.host_key_fingerprint}",
  "authenticationType": "SshPublicKey",
  "userName": "${var.user_name}",
  "privateKeyContent": {
    "type": "AzureKeyVaultSecret",
    "store": {
      "referenceName": "${var.keyvault_name}",
      "type": "LinkedServiceReference"
    },
    "secretName": "${var.keyvault_secret_key}"
  },
  "passPhrase": {
    "type": "AzureKeyVaultSecret",
    "store": {
      "referenceName": "${var.keyvault_name}",
      "type": "LinkedServiceReference"
    },
    "secretName": "${var.keyvault_secret_passphrase}"
  }
}
JSON
}