search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.59k stars 4.63k forks source link

Support for azurerm_key_vault_encrypted_value export plain_text_value #20763

Closed EZhao-Quest closed 1 year ago

EZhao-Quest commented 1 year ago

Is there an existing issue for this?

Community Note

d

Description

azurerm_key_vault_encrypted_value should return plain_text_value when I pass a encrypted_data .

user case:

we want to use terraform to auto create a azure keyvault secret to store senstive data.

we cannot put the plain text in our source code, we need put an encrpted string to the source code in github.

the expected behaviour is we put the encrpted data to data.azurerm_key_vault_encrypted_value, then get the plain text from data.azurerm_key_vault_encrypted_value.plain_text_value, pass to the value of azurerm_key_vault_secret

current azurerm_key_vault_encrypted_value only return an id. cannot get plain_text_value.

New or Affected Resource(s)/Data Source(s)

azurerm_key_vault_encrypted_value

Potential Terraform Configuration

data "azurerm_key_vault" "default" {
  name                = "mytest-keyvault"
  resource_group_name = "mytest-rg"
}

data "azurerm_key_vault_key" "default" {
  name         = "my-test-key"
  key_vault_id = data.azurerm_key_vault.default.id
}

resource "azurerm_key_vault_secret" "default" {
  name         = "test_secret"
  key_vault_id = data.azurerm_key_vault.default.id
  value        = data.azurerm_key_vault_encrypted_value.decrypted.plain_text_value
}

data "azurerm_key_vault_encrypted_value" "decrypted" {
  key_vault_key_id = data.azurerm_key_vault_key.default.id
  algorithm        = "RSA1_5"
  encrypted_data   = var.encrypted_secret_string
}

the encrypted_secret_string get from: az keyvault key encrypt --name xxx_keyname --vault-name xxx_key_vault_name --algorithm RSA1_5 --value "adfs azure" --data-type plaintext

References

No response

wuxu92 commented 1 year ago

HI @EZhao-Quest i have tried with you config and found that you can get the plain_text_value in fact if I understand you correct.

image

EZhao-Quest commented 1 year ago

@wuxu92 yes, use nonsenstive can get th data. but there is a case, my test string include space, like this "this is a test", the result missing space, return the value "thisisatsg". image

EZhao-Quest commented 1 year ago

I also tried convert "this is a test" to base64 string, then get encrypted_data, put the encrypted_data to azurerm_key_vault_encrypted_value.decrypted block. the output missing padding, like this "dGhpcyBpcyBhIHRlc3Q", the expected value should be "dGhpcyBpcyBhIHRlc3Q="

image

image

pellepelster commented 1 year ago

I am also affected by this issue, to me it looks like a padding problem somewhere, see this minimal example:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=3.45.0"
    }
  }
}

provider "azurerm" {
  features {}
}

data "azurerm_client_config" "current" {
}

resource azurerm_resource_group "test" {
  name     = "test"
  location = "westeurope"
}

resource "random_string" "random" {
  length  = 8
  special = false
  upper   = false
}

resource "azurerm_key_vault" "test" {
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  name                = "test-${random_string.random.id}"
  sku_name            = "standard"
  tenant_id           = data.azurerm_client_config.current.tenant_id
}

resource "azurerm_key_vault_key" "test" {
  key_opts = [
    "decrypt",
    "encrypt",
    "sign",
    "unwrapKey",
    "verify",
    "wrapKey",
  ]

  key_type     = "RSA"
  key_vault_id = azurerm_key_vault.test.id
  name         = "test"
  key_size     = 2048
}

data "azurerm_key_vault_encrypted_value" "encrypt" {
  key_vault_key_id = azurerm_key_vault_key.test.id
  algorithm        = "RSA1_5"
  plain_text_value = "some-secret"
}

data "azurerm_key_vault_encrypted_value" "decrypt" {
  key_vault_key_id = azurerm_key_vault_key.test.id
  algorithm        = "RSA1_5"
  encrypted_data   = data.azurerm_key_vault_encrypted_value.encrypt.encrypted_data
}

output "encrypted" {
  value = nonsensitive(data.azurerm_key_vault_encrypted_value.encrypt.encrypted_data)
}

output "decrypted" {
  value = nonsensitive(data.azurerm_key_vault_encrypted_value.decrypt.plain_text_value)
}

the output looks like this:

decrypted = "some-secres"
encrypted = "UARxWKJtEPzrYjQnGL4o8BxfJWI8FIoHCiEWnzk06et4yWwN1vovKRwhTT4XBMp7KkFnORQp4rTn_8V8wouWjdgpCIKaDcOTEDR7dBPnsg2Wj1NW-5rhLIo9BsZsdZ2auTQbzx5_6zz8LR-TGLFmdBjkoIt19AAeKaoDNA8R3cP09bRfmjN45xCS1RbutRgJNOu2U49YESXohcKEIq3PVR91dxUeevxdzs3rgxbK-UJfu-n9HTKtbYFh7jG8NBtlFnpD9KxeIy8NOfTyQ8Nqk6xYPtrU4LAGeMvftC79lKw6ahP9tV8z75YZTKmhPHcL9Wg26Jy-CK39tufdE3gFmA"

while experimenting with creation of secrets from already encrypted secrets I saw that the end of the base64 somehow seems to be mangled sometimes. That would be in line with the observation of the mangled last character of the secret with the output.

pellepelster commented 1 year ago

here is an integration test highlighting the issue: https://github.com/hashicorp/terraform-provider-azurerm/compare/main...pellepelster:terraform-provider-azurerm:main

pellepelster commented 1 year ago

I am not sure how the encrypt API of KeyVault is intended to be used, but this script using the plain REST API produces the same results:

#!/usr/bin/env bash

set -euo pipefail

ARM_SUBSCRIPTION_ID="xxx"
ARM_CLIENT_ID="xxx"
ARM_CLIENT_SECRET="xxx"
ARM_TENANT_ID="xxx"

VAULT_BASE_URL="https://xxx.vault.azure.net"
VAULT_KEY_NAME="xxx"
SECRET="some-secret"

ACCES_TOKEN=$(curl --silent -X POST -d "grant_type=client_credentials&client_id=${ARM_CLIENT_ID}&client_secret=${ARM_CLIENT_SECRET}&resource=https%3A%2F%2Fvault.azure.net" https://login.microsoftonline.com/${ARM_TENANT_ID}/oauth2/token | jq -r ".access_token")

ENCRYPT_RESPONSE=$(curl --silent -X POST -d "{ \"alg\": \"RSA1_5\", \"value\": \"${SECRET}\" }" -H "Authorization: Bearer ${ACCES_TOKEN}" -H "Content-Type: application/json" "${VAULT_BASE_URL}/keys/${VAULT_KEY_NAME}/encrypt?api-version=7.4")

ENCRYPTED=$(echo "${ENCRYPT_RESPONSE}" | jq -r ".value")

DECRYPT_RESPONSE=$(curl --silent -X POST -d "{ \"alg\": \"RSA1_5\", \"value\": \"${ENCRYPTED}\" }" -H "Authorization: Bearer ${ACCES_TOKEN}" -H "Content-Type: application/json" "${VAULT_BASE_URL}/keys/${VAULT_KEY_NAME}/decrypt?api-version=7.4" | jq)

DECRYPTED=$(echo "${DECRYPT_RESPONSE}" | jq -r ".value")

echo "secret was: '${SECRET}'"
echo "after encryption/decryption: '${DECRYPTED}'"

in my case:

secret was: 'some-secret'
after encryption/decryption: 'some-secres'
wuxu92 commented 1 year ago

thanks @pellepelster for your informatin, i reproduced it and have the same result as yours. i have filed an issue in Azure-rest-api-spec project to see if the service team has any suggestions.

dalzhao commented 1 year ago

in my case:

secret was: 'some-secret'
after encryption/decryption: 'some-secres'

Hi @pellepelster ,

Key Vault uses Base64 URL encoding for the content to be encrypted / decrypted.

Because each character in the base64 encoding represents 64 possible cases, and 2^6 = 64, so each base64 character encodes 6 bits of data, which is not always aligned with the 8-bit byte we are working with more directly.

In your case, "some-secres" "some-secret" "some-secreu" and "some-secrev" all represent the same byte array.

PS C:\Users\dalzhao> [convert]::FromBase64String("some+secres=") | Format-Hex

           Path:

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   B2 89 9E FA C7 9C AD EB                          ²úÇ­ë

PS C:\Users\dalzhao> [convert]::FromBase64String("some+secret=") | Format-Hex

           Path:

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   B2 89 9E FA C7 9C AD EB                          ²úÇ­ë

PS C:\Users\dalzhao> [convert]::FromBase64String("some+secreu=") | Format-Hex

           Path:

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   B2 89 9E FA C7 9C AD EB                          ²úÇ­ë

PS C:\Users\dalzhao> [convert]::FromBase64String("some+secrev=") | Format-Hex

           Path:

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   B2 89 9E FA C7 9C AD EB                          ²úÇ­ë

The 11 characters in "some-secret" contains 66 bits of information, but we only encode 64 bits or 8-byte of information, the extra 2 bit of information is redundant, allowing us to use any of the "stuv" as the last character.

If you want to encrypt "some-secret", you need to do the following.

  1. Convert "some-secret" from string to byte array. 

    
PS C:\Users\dalzhao> $bytes = [System.Text.Encoding]::UTF8.GetBytes("some-secret")
PS C:\Users\dalzhao> $bytes | Format-Hex

       Path:

       00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000 73 6F 6D 65 2D 73 65 63 72 65 74 some-secret


2. Convert the byte-array into base64 url string

PS C:\Users\dalzhao> [convert]::ToBase64String($bytes) -replace [regex]::Escape('+'),'-' -replace '/','_' -replace '=' c29tZS1zZWNyZXQ



3. send it through the encrypt API.
dalzhao commented 1 year ago

@EZhao-Quest , Azure Key Vault uses base64-URL encoding in the response.

To convert base64 URL encoding back to base64 encoding, we need to replace the '-' with '+', '_', with '/', and then add either 1 or 2 padding characters to make the length a multiple of 4.

wuxu92 commented 1 year ago

@EZhao-Quest , Azure Key Vault uses base64-URL encoding in the response.

To convert base64 URL encoding back to base64 encoding, we need to replace the '-' with '+', '_', with '/', and then add either 1 or 2 padding characters to make the length a multiple of 4.

@dalzhao does this mean we have to add the padding characters in client side?

another point is that passing a string that is not base64url encoded to encrypt API is also acceptable and then the result of decrypt API is not base64url encoded too? just like use some-secret as the value of encrypt.

pellepelster commented 1 year ago

@dalzhao

Thanks for the detailed explanation. That is completely understandable from a technical point of view (and also in line with other solutions like AWS KMS or Hashicorp Vault). I think it would be beneficial to be a little more explicit about this in the docs (https://learn.microsoft.com/en-us/rest/api/keyvault/keys/encrypt/encrypt?tabs=HTTP) because value is just described as string there.

Even more confusing in the sourcecode it is marked as base64 encoded string

image

dalzhao commented 1 year ago

@wuxu92, you have 2 points:

@dalzhao does this mean we have to add the padding characters in client side?

Yes. We don't expect customers to call the REST API directly. We expect the applications calling Azure Key Vault would take advantage of the Azure Key Vault SDK library, which should handle all the conversions.

another point is that passing a string that is not base64url encoded to encrypt API is also acceptable and then the result of decrypt API is not base64url encoded too? just like use some-secret as the value of encrypt.

I don't quite understand your comment. "some-secret" is a valid base-64 url encoded string. The 64 characters you can use in a base64 url encoded string are A-Z, a-z, 0-9, - (hyphen) and _ (underscore).

wuxu92 commented 1 year ago

I don't quite understand your comment. "some-secret" is a valid base-64 url encoded string. The 64 characters you can use in a base64 url encoded string are A-Z, a-z, 0-9, - (hyphen) and _ (underscore).

thanks for your explanation, i understand now for some-secret is a valid base64url encoded string.

We expect the applications calling Azure Key Vault would take advantage of the Azure Key Vault SDK library, which should handle all the conversions.

terraform use the go-azure-sdk to call the API directly so we have to handle encoding/decoding manually.

thanks again for your help!

github-actions[bot] commented 5 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.