Vault Rekey fails for transit-unseal cluster

Vault rekey operation fails in an automatic unseal cluster when unsealed with transit key, most likely transit-token despite being refreshed, loose ability to rekey when initial TTL (30days) is exceeded. Generation of the new transit token allows to perform rekey again.

To Reproduce (not tested yet):

Create transit token with short lifetime
perform vault rekey
wait until token expire, check with vault token lookup if that token was refreshed
Perform rekey, observe how it failed
Generate new transit token, update configuration, perform rekey again - should work.

Observed steps

Vault operator rekey (last step) with the transit automatic-unseal cluster:

echo $VAULT_UNSEAL_KEY | /usr/local/bin/vault operator rekey -format json -target=recovery -nonce 79d2de9b-877e-ffac-284b-xxxx - 

    Error posting unseal key: Error making API request.

    URL: PUT https://vault-master-1-devtest.atest.aval:8200/v1/sys/rekey-recovery-key/update
    Code: 400. Errors:

    * recovery key verification failed: failed to decrypt encrypted stored keys: Error making API request.

    URL: PUT https://vault.devtest-avalaunch.aval:8200/v1/transit/decrypt/autounseal
    Code: 403. Errors:

    * permission denied

Transit token in the Vault Shamir (manual unseal), s.nOA4acDIvIYcCqTXsedBVxxx is a transit token. As you can see, it is being renewed correctly (but stays constant in the vault config file).

[vault@vault-1-devtest ~]$ vault token lookup s.nOA4acDIvIYcCqTXsedBVxxx
Key                  Value
---                  -----
accessor             jAmBvAcn1rkrUAnnEQTSHyyy
creation_time        1613218755
creation_ttl         768h
display_name         token
entity_id            n/a
expire_time          2021-04-14T15:29:47.449517197+03:00
explicit_max_ttl     0s
id                   s.nOA4acDIvIYcCqTXsedBVxxx
issue_time           2021-02-13T14:19:15.726240986+02:00
last_renewal         2021-03-13T14:29:47.449517309+02:00
last_renewal_time    1615638587
meta                 <nil>
num_uses             0
orphan               false
path                 auth/token/create
period               768h
policies             [autounseal default]
renewable            true
ttl                  766h54m44s
type                 service

Vault Shamir (manual unseal)

/usr/local/bin/vault token create -policy="autounseal" -format=json
{ 
  "request_id": "f42cd31d-fdd3-a641-6af9-zzzzz",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": null,
  "warnings": null,
  "auth": {
    "client_token": "s.xxxxxxxxxxxxxxx",
    "accessor": "yyyyyy",
    "policies": [
      "autounseal",
      "default"
    ],
    "token_policies": [
      "autounseal",
      "default"
    ],
    "identity_policies": null,
    "metadata": null,
    "orphan": false,
    "entity_id": "",
    "lease_duration": 2764800,
    "renewable": true
  }
}

I had to generate new token, put it into configuration. After that, using new token with same policies allows for rekey in the auto-unseal cluster.

Rekey works after updating transit token in auto-useal cluster with the 'fresh one'.

Expected behavior Rekey works as long as the transit token is refreshed :)

Environment:

Vault Server Version (retrieve with vault status): v1.4.2
Vault CLI Version (retrieve with vault version): v1.4.2
Server Operating System/Architecture: RedHat 7

Vault server configuration file(s):

auto-unseal cluster config

cluster_name = "dc2"
max_lease_ttl = "768h"
default_lease_ttl = "768h"

disable_clustering = "False"
cluster_addr = "https://vault-master-1-dev.cc.cc:8201"
api_addr = "https://vault-master.dev-cc.cc:8200"

plugin_directory = "/usr/local/lib/vault/plugins"

listener "tcp" {
  address = "vault-master-1-dev.cc.cc:8200"
  cluster_address = "vault-master-1-dev.cc.cc:8201"
  tls_cert_file = "/var/vault/vault-master-cc.pem"
  tls_key_file = "/var/vault/vault-master-cc.key"
  tls_client_ca_file = "/var/vault/ca.cer"
  tls_min_version = "tls12"
  tls_prefer_server_cipher_suites = "false"
  tls_disable = "false"
}

storage "raft" {
  path = "/var/vault"
  node_id = "4"
  }

ui = true
seal "transit" {
  address = "https://vault.dev-cc.cc:8200"
  disable_renewal = "false"
  key_name = "autounseal"
  mount_path = "transit/"
  token = "s.cccc"
  tls_skip_verify = "false"
  tls_ca_cert = "/var/vault/ca.cer"
  tls_client_cert = "/var/vault/vault-master-cc.pem"
  tls_client_key = "/var/vault/vault-master-cc.key"

My guess here is: the token is refreshed up to default_lease_ttl, as long as it is being used OR up to explicit_max_ttl if it is used.

When the token expire, it stops working. When the token is not refreshed (eg because vault is down, noone uses it) -> it expires -> it is not working.

It looks to me, that even the short live token without explicit_max_ttl would be refreshed almost indefinitely as long as the vault is working, but (here is my guess) after max_lease_ttl = "768h" it will fail to regenerate recovery key (rekey)

Example of the token that was in use in automatic-unseal cluster, then was replaced by new token, and after 15 minutes it died (so was working for 1,5h). During that time the rekey was available, and a cluster node restart was successful, node was unsealed automatically.

Fri Mar 19 17:01:49 EET 2021

[vault@vault-1-dev ~]$ vault token lookup s.YL5jvpyYHGUPowOk5rvHHnhR
Key                  Value
---                  -----
accessor             31mN9uQxfv6TKcPljUB5GC4j
creation_time        1616162289
creation_ttl         15m
display_name         token
entity_id            n/a
expire_time          2021-03-19T17:18:33.560988024+02:00
explicit_max_ttl     0s
id                   s.YL5jvpyYHGUPowOk5rvHHnhR
issue_time           2021-03-19T15:58:09.603239323+02:00
last_renewal         2021-03-19T17:03:33.560988144+02:00
last_renewal_time    1616166213
meta                 <nil>
num_uses             0
orphan               false
path                 auth/token/create
policies             [autounseal default]
renewable            true
ttl                  9m50s
type                 service

[vault@vault-1-dev ~]$ date
Fri Mar 19 17:08:51 EET 2021

[vault@vault-1-dev ~]$ sleep 530; vault token lookup s.YL5jvpyYHGUPowOk5rvHHnhR;
Error looking up token: Error making API request.

URL: POST https://vault-1-dev.atest.aval:8200/v1/auth/token/lookup
Code: 403. Errors:

* bad token
Error looking up token: Error making API request.

in contrast to that, a token with *explicit_max_ttl=15m died while being in use by the auto-unseal cluster, it could not be refreshed more that the TTL

[vault@vault-1-dev ~]$ /usr/local/bin/vault token create -policy="autounseal" -ttl=5m -explicit-max-ttl=15m 
Key                  Value
---                  -----
token                s.L3487xFUTvlnpUOorGxTgjxr
token_accessor       u10jdJrAvr5gHOCnhg2XjMxW
token_duration       5m
token_renewable      true
token_policies       ["autounseal" "default"]
identity_policies    []
policies             ["autounseal" "default"]

[vault@vault-1-dev ~]$ /usr/local/bin/vault token  lookup  s.L3487xFUTvlnpUOorGxTgjxr
Key                 Value
---                 -----
accessor            u10jdJrAvr5gHOCnhg2XjMxW
creation_time       1616161121
creation_ttl        5m
display_name        token
entity_id           n/a
expire_time         2021-03-19T15:43:41.198130709+02:00
explicit_max_ttl    15m
id                  s.L3487xFUTvlnpUOorGxTgjxr
issue_time          2021-03-19T15:38:41.198141289+02:00
meta                <nil>
num_uses            0
orphan              false
path                auth/token/create
policies            [autounseal default]
renewable           true
ttl                 4m38s
type                service

[vault@vault-1-dev ~]$  /usr/local/bin/vault token  lookup  s.L3487xFUTvlnpUOorGxTgjxr
Error looking up token: Error making API request.

URL: POST https://vault-1-dev.atest.aval:8200/v1/auth/token/lookup
Code: 403. Errors:

* bad token

hashicorp / vault

Vault Rekey fails for transit-unseal cluster #11149