Venafi Plugin for Hashicorp throwing TTL Errors

[Tony-venafi]

PROBLEM SUMMARY Receiving TTL errors while using the monitor plugin. Original issue was " we can issue the cert, sign it, but can't save it in Venafi. also getting some strange warnings logged: (TTL of \"2159h59m59.500822605s\" exceeded the effective max_ttl of \"768h\"; TTL value is capped accordingly) As if it is trying to cap on TTL of logged in identity.}}"

This seemed to be resolved by editing the settings on Vault and changing the ttl value. The error no longer showed. However the new value is throwing a different error.

Now, receiving "Now ttl=”100h” term is giving an error ("error":"1 error occurred:\n\t* permission denied\n\n"}) "

STEPS TO REPRODUCE Change the ttl on vault to 100h. This TPP environment is hosted in Azure. Exact reproduction steps are unknown, as we only know what they are receiving when setting the TTL to 100h. TPP version 20.2 Vault plugin version: 0.7.1

EXPECTED RESULTS Expect to see regular issuance of certificates and not be getting these errors within the plugin. We know that the Venafi plugin lacks the functionality to consider the TTL settings from vault anyway, but wondering why we are getting this new permission denied error.

ACTUAL RESULTS Now ttl=”100h” term is giving an error ("error":"1 error occurred:\n\t* permission denied\n\n"})

ENVIRONMENT DETAILS This is an Azure environment. TPP version is 20.2 Vault plugin version is 0.7.1.

COMMENTS/WORKAROUNDS

[tr1ck3r]

@Tony-venafi Based on the description you've provided I believe you meant to open this for the https://github.com/Venafi/vault-pki-backend-venafi. If so, would you please submit the issue there?

[Tony-venafi]

Apologies. I have opened this on the backend plugin.

Thanks

[tr1ck3r]

Thank you!

[tr1ck3r]

After re-reading the details provided and gathering some additional information out-of-band, I believe this is in fact referring to the Vault Monitor and not the Vault Backend. The Vault server TTL was increased to address "exceed the effective max_ttl" warnings that were being logged when enrolling certificates using the Vault Backend because the Vault server TTL (768h = 32 days) was much less than the validity of the certificates being issued by CA assigned to the Venafi zone (2160h = 90 days).

[tr1ck3r]

Confirmed plugin version is actually v0.6.0 (latest release).

@Tony-venafi can you please provide the vault server config default_lease_ttl and max_lease_ttl both before (when the Vault Monitor was working) and now? Also please confirm what version of Vault is being used and if any ttl or max_ttl values were specified when the role was created. It's not immediately apparent how changing the TTL would affect permissions related to our secrets engine. Is there anything more written to the Vault log when the error occurs?

[Tony-venafi]

Missed the update here.

Forwarded questions to the customer.

[Tony-venafi]

---vault config----

max_lease_ttl = "87600h"

disable_performance_standby = true

ui = true

storage "consul" {

address = "127.0.0.1:8500"

path = "vault"

token = "*****"

}

listener "tcp" {

address = "0.0.0.0:8200"

tls_disable = false

tls_cert_file = "/etc/vault.d/vault.crt"

tls_key_file = "/etc/vault.d/vault.key"

}

listener "tcp" {

address = "0.0.0.0:443"

tls_disable = false

tls_cert_file = "/etc/vault.d/vault.crt"

tls_key_file = "/etc/vault.d/vault.key"

}

seal "azurekeyvault" {

tenant_id = "56c62bbe**"

vault_name = "vaulteastus2t**"

key_name = "vault-eastus2-test-c2-seal"

}

api_addr = "https://c2-test-azure-eastus2.vault.*******.com"

plugin_directory = "/etc/vault/plugins"

cluster_addr = "https://10.210.1.82:8201"

-------------vault config ends-------------

Only change before and after is

Before we were not having ln#1 in vault config (max_lease_ttl = "87600h"), so vault was assuming default max_lease_ttl of 768h)

---

2. Also please confirm what version of Vault is being used and if any ttl or max_ttl values were specified when the role was created?

Before and after vault version was same 1.5.1+prem

---

3. Is there anything more written to the Vault log when the error occurs?

Detailed request response log

vault write testns/pki/***-internal/issue/secretsmanagement.17405 common_name="app-abc-test.vaultpoc.****.com" ttl="1h"

Sep 16 15:52:56 vault-eastus2-test-c2-vault00001A vault-audit[11894]: {"time":"2020-09-16T15:52:56.328498212Z","type":"request","auth":{"client_token":"hmac-sha256:79cfbc42e3c01a47c160d84f46089d45da13378068d2a251bb5d20bc89338647","accessor":"hmac-sha256:ed8a4f29550e14f7f98fc6875235e0955771c8775c17a7126e7dd2f817f98502","display_name":"ldap-rxj0044","policies":["admin_policy","default","gaia.12345_policy","gaia.12346_policy","pilot.00000_policy","provisioner_policy","secureassist.16762_policy","tap.12955_policy","tap.13447_policy","tdt.15182_policy","vurv.13673_policy"],"token_policies":["admin_policy","default","gaia.12345_policy","gaia.12346_policy","pilot.00000_policy","provisioner_policy","secureassist.16762_policy","tap.12955_policy","tap.13447_policy","tdt.15182_policy","vurv.13673_policy"],"metadata":{"username":"***"},"entity_id":"d22a83fc-41d2-e9c8-707a-50843eb255d5","token_type":"service","token_ttl":2764800,"token_issue_time":"2020-09-16T15:52:03Z"},"request":{"id":"bd1c4f8e-8d95-5380-43c3-18561b26c20a","operation":"update","mount_type":"vault-pki-monitor-venafi_strict","client_token":"hmac-sha256:79cfbc42e3c01a47c160d84f46089d45da13378068d2a251bb5d20bc89338647","client_token_accessor":"hmac-sha256:ed8a4f29550e14f7f98fc6875235e0955771c8775c17a7126e7dd2f817f98502","namespace":{"id":"5tuB8","path":"testns/"},"path":"pki/**-internal/issue/secretsmanagement.17405","data":{"common_name":"hmac-sha256:e9c98b883bdb302a9f32dffe3b930d88275e5869621efa253ea055fd4510ae59","ttl":"hmac-sha256:d46c5b780bb550e87aff7d6427fbf5a2203af47648a13eda15c1e762da4a2b25"},"remote_address":"127.0.0.1"},"error":"1 error occurred:\n\t* permission denied\n\n"}

Sep 16 15:52:56 vault-eastus2-test-c2-vault00001A vault-audit[11894]: {"time":"2020-09-16T15:52:56.328778526Z","type":"response","auth":{"client_token":"hmac-sha256:79cfbc42e3c01a47c160d84f46089d45da13378068d2a251bb5d20bc89338647","accessor":"hmac-sha256:ed8a4f29550e14f7f98fc6875235e0955771c8775c17a7126e7dd2f817f98502","display_name":"ldap-rxj0044","policies":["admin_policy","default","gaia.12345_policy","gaia.12346_policy","pilot.00000_policy","provisioner_policy","secureassist.16762_policy","tap.12955_policy","tap.13447_policy","tdt.15182_policy","vurv.13673_policy"],"token_policies":["admin_policy","default","gaia.12345_policy","gaia.12346_policy","pilot.00000_policy","provisioner_policy","secureassist.16762_policy","tap.12955_policy","tap.13447_policy","tdt.15182_policy","vurv.13673_policy"],"metadata":{"username":"**"},"entity_id":"d22a83fc-41d2-e9c8-707a-50843eb255d5","token_type":"service","token_ttl":2764800,"token_issue_time":"2020-09-16T15:52:03Z"},"request":{"id":"bd1c4f8e-8d95-5380-43c3-18561b26c20a","operation":"update","mount_type":"vault-pki-monitor-venafi_strict","client_token":"hmac-sha256:79cfbc42e3c01a47c160d84f46089d45da13378068d2a251bb5d20bc89338647","client_token_accessor":"hmac-sha256:ed8a4f29550e14f7f98fc6875235e0955771c8775c17a7126e7dd2f817f98502","namespace":{"id":"5tuB8","path":"testns/"},"path":"pki/**-internal/issue/secretsmanagement.17405","data":{"common_name":"hmac-sha256:e9c98b883bdb302a9f32dffe3b930d88275e5869621efa253ea055fd4510ae59","ttl":"hmac-sha256:d46c5b780bb550e87aff7d6427fbf5a2203af47648a13eda15c1e762da4a2b25"},"remote_address":"127.0.0.1"},"response":{"mount_type":"vault-pki-monitor-venafi_strict","data":{"error":"hmac-sha256:0f2618cd41155cde830944871def743cfabba383fc661a96d8568ebb16c47f3e"}},"error":"1 error occurred:\n\t* permission denied\n\n"}

Now if I remove ttl, it works

vault write testns/pki/**-internal/issue/secretsmanagement.17405 common_name="app-abc-test.vaultpoc.**.com"

Sep 16 15:54:00 vault-eastus2-test-c2-vault00001A vault-audit[11894]: {"time":"2020-09-16T15:54:00.477991073Z","type":"request","auth":{"client_token":"hmac-sha256:79cfbc42e3c01a47c160d84f46089d45da13378068d2a251bb5d20bc89338647","accessor":"hmac-sha256:ed8a4f29550e14f7f98fc6875235e0955771c8775c17a7126e7dd2f817f98502","display_name":"ldap-rxj0044","policies":["admin_policy","default","gaia.12345_policy","gaia.12346_policy","pilot.00000_policy","provisioner_policy","secureassist.16762_policy","tap.12955_policy","tap.13447_policy","tdt.15182_policy","vurv.13673_policy"],"token_policies":["admin_policy","default","gaia.12345_policy","gaia.12346_policy","pilot.00000_policy","provisioner_policy","secureassist.16762_policy","tap.12955_policy","tap.13447_policy","tdt.15182_policy","vurv.13673_policy"],"metadata":{"username":"***"},"entity_id":"d22a83fc-41d2-e9c8-707a-50843eb255d5","token_type":"service","token_ttl":2764800,"token_issue_time":"2020-09-16T15:52:03Z"},"request":{"id":"775c0cb1-d4a5-4a88-d4e0-14e62ee20172","operation":"update","mount_type":"vault-pki-monitor-venafi_strict","client_token":"hmac-sha256:79cfbc42e3c01a47c160d84f46089d45da13378068d2a251bb5d20bc89338647","client_token_accessor":"hmac-sha256:ed8a4f29550e14f7f98fc6875235e0955771c8775c17a7126e7dd2f817f98502","namespace":{"id":"5tuB8","path":"testns/"},"path":"pki/**-internal/issue/secretsmanagement.17405","data":{"common_name":"hmac-sha256:e9c98b883bdb302a9f32dffe3b930d88275e5869621efa253ea055fd4510ae59"},"remote_address":"127.0.0.1"}}

Sep 16 15:54:00 vault-eastus2-test-c2-vault00001A vault-audit[11894]: {"time":"2020-09-16T15:54:00.667452305Z","type":"response","auth":{"client_token":"hmac-sha256:79cfbc42e3c01a47c160d84f46089d45da13378068d2a251bb5d20bc89338647","accessor":"hmac-sha256:ed8a4f29550e14f7f98fc6875235e0955771c8775c17a7126e7dd2f817f98502","display_name":"ldap-rxj0044","policies":["admin_policy","default","gaia.12345_policy","gaia.12346_policy","pilot.00000_policy","provisioner_policy","secureassist.16762_policy","tap.12955_policy","tap.13447_policy","tdt.15182_policy","vurv.13673_policy"],"token_policies":["admin_policy","default","gaia.12345_policy","gaia.12346_policy","pilot.00000_policy","provisioner_policy","secureassist.16762_policy","tap.12955_policy","tap.13447_policy","tdt.15182_policy","vurv.13673_policy"],"metadata":{"username":"***"},"entity_id":"d22a83fc-41d2-e9c8-707a-50843eb255d5","token_type":"service","token_ttl":2764800,"token_issue_time":"2020-09-16T15:52:03Z"},"request":{"id":"775c0cb1-d4a5-4a88-d4e0-14e62ee20172","operation":"update","mount_type":"vault-pki-monitor-venafi_strict","client_token":"hmac-sha256:79cfbc42e3c01a47c160d84f46089d45da13378068d2a251bb5d20bc89338647","client_token_accessor":"hmac-sha256:ed8a4f29550e14f7f98fc6875235e0955771c8775c17a7126e7dd2f817f98502","namespace":{"id":"5tuB8","path":"testns/"},"path":"pki/**-internal/issue/secretsmanagement.17405","data":{"common_name":"hmac-sha256:e9c98b883bdb302a9f32dffe3b930d88275e5869621efa253ea055fd4510ae59"},"remote_address":"127.0.0.1"},"response":{"mount_type":"vault-pki-monitor-venafi_strict","secret":{"lease_id":"pki/**-internal/issue/secretsmanagement.17405/UTkvbXtuyropXYhLlTLtUhPy.5tuB8"},"data":{"ca_chain":["hmac-sha256:10ef62e84bd3096dbd7a1d5a1fc6b03abee0f24467d3178598b1e7ac47e1e250"],"certificate":"hmac-sha256:276f6aa353712ff7c0d86ec9b242d97608ca49530cb2515332753ad7065f82b1","expiration":1608047640,"issuing_ca":"hmac-sha256:10ef62e84bd3096dbd7a1d5a1fc6b03abee0f24467d3178598b1e7ac47e1e250","private_key":"hmac-sha256:0fae5eae6cc6cc7041e27bb066912b4309ff2f199dc09cea0a9ea13de5f65728","private_key_type":"hmac-sha256:59b297480ddc54d0bddda1f8123c2554f970ee5fbc6d40e0c64cc92c8e563807","serial_number":"hmac-sha256:dfc8fa91de2d92eb7d025472319a18daca4231516ad82562a5faf15fa29644f8"}}}

[tr1ck3r]

@Tony-venafi So far unable to reproduce the issue using the same versions of Vault and Venafi plugins in Performance Replication architecture and with the same TTL values applied at the server, role, and request levels. The only way I can make sense of the reported result is a security policy that doesn't allow the user executing the vault write testns/pki/******-internal/issue/secretsmanagement.17405 command to specify the ttl parameter. See "allowed_parameters" and "denied_parameters" in https://www.vaultproject.io/docs/concepts/policies#parameter-constraints. Is there any security policy like that applied? Are you able to execute the command successfully using a root token? Even if the security policy looks correct, perhaps reapplying it will resolve the issue?

[Tony-venafi]

"generate_lease": true, "max_ttl": 7776000, "no_store": false, "ttl": 7776000, "venafi_check_policy": "secretsmanagement.17405", "venafi_import": true, "venafi_import_timeout": 15, "venafi_import_workers": 1,

[Tony-venafi]

I have submitted those questions to the customer.

[tr1ck3r]

@Tony-venafi any update? Was the security policy the issue?

[tr1ck3r]

Got confirmation the root cause of this issue was the Vault security policy which was configured to forbid the ttl parameter. The customer was able to resolve the problem by adding "*" to the allowed_parameters for the secrets engine path.