Vault data not getting updated in the secondary datacenter due to the cache being permanent

[alekhrj]

Describe the bug Setup/Deployment : multi DC vault setup with consul as the storage backend and the consul data is being replicated using consul-replicate.

Bug : when data in the vault is updated in the primary datacenter, updated value is not reflected in the response from the secondary datacenter. This is because the data cached in the vault is almost permanent.

To Reproduce Steps to reproduce the behavior: Create a key (transit engine) in the primary datacenter

1. Run vault write -f transit/keys/alekh-test-13

Read the key in the secondary datacenter (after successful replication)

2. Run ./vault read transit/keys/alekh-test-13

   Key                       Value
   ---                       -----
   allow_plaintext_backup    false
   auto_rotate_period        0s
   deletion_allowed          false
   derived                   false
   exportable                false
   imported_key              false
   keys                      map[1:1670324168]
   latest_version            1
   min_available_version     0
   min_decryption_version    1
   min_encryption_version    0
   name                      alekh-test-13
   supports_decryption       true
   supports_derivation       true
   supports_encryption       true
   supports_signing          false
   type                      aes256-gcm96

Update any property (deletion_allowed here) of the key in the primary datacenter

3. Run curl --header "X-Vault-Token: <valid token>" --request POST --data @payload.json http://127.0.0.1:8200/v1/transit/keys/alekh-test-13/config

payload.json : https://developer.hashicorp.com/vault/api-docs/secret/transit#sample-payload-3

Read the key in the secondary datacenter (after successful replication)

4. Run ./vault read transit/keys/alekh-test-13

   Key                       Value
   ---                       -----
   allow_plaintext_backup    false
   auto_rotate_period        0s
   deletion_allowed          false
   derived                   false
   exportable                false
   imported_key              false
   keys                      map[1:1670324168]
   latest_version            1
   min_available_version     0
   min_decryption_version    1
   min_encryption_version    0
   name                      alekh-test-13
   supports_decryption       true
   supports_derivation       true
   supports_encryption       true
   supports_signing          false
   type                      aes256-gcm96

Expected behavior In step-4, we get property value deletion_allowed=false, instead of true

New value is updated only after the leader election when cache is rebuilt.

Similar behavior is observed for other backends as well.

Environment:

• Vault Server Version (retrieve with vault status):

  
  # ./vault status
  Key             Value
  ---             -----
  Seal Type       shamir
  Initialized     true
  Sealed          false
  Total Shares    1
  Threshold       1
  Version         1.13.0-dev1
  Build Date      n/a
  Storage Type    consul
  Cluster Name    vault-cluster-67dc2e08
  Cluster ID      7c42e0a1-8be9-59d5-2ec5-2032a2b9be6a
  HA Enabled      true
  HA Cluster      https://10.51.114.216:8201
  HA Mode         active
  Active Since    2022-12-06T10:01:12.402497608Z

* Vault CLI Version (retrieve with `vault version`):
Vault v1.13.0-dev1

* Server Operating System/Architecture:

cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" VERSION_CODENAME=stretch ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Vault server configuration file(s):

```hcl
# primary DC
{
    "ui": true,
  "telemetry": {
    "statsd_address": "127.0.0.1:8125"
  },
  "storage": {
    "consul": {
      "address": "127.0.0.1:8500",
      "path": "vault",
      "obfuscate_paths": 1,
      "token": "e98xxxxx-8xxx-8xx5-xxx6-d2xxxa5f9xxx"
    }
  },
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
 },
  "plugin_directory": "/var/lib/fk-sec-vault/plugins"
}

# secondary DC
{
    "disable_cache": false,
    "ui": true,
  "telemetry": {
    "statsd_address": "127.0.0.1:8125"
  },
  "storage": {
    "consul": {
      "address": "127.0.0.1:8500",
      "path": "vault",
      "obfuscate_paths": 1,
      "token": "e98xxxxx-8xxx-8xx5-xxx6-d2xxxa5f9xxx",
    "consistency_mode": "strong",
    "cache_size": 1310720
    }
  },
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
 },
  "plugin_directory": "/var/lib/fk-sec-vault/plugins"
}

[maxb]

This issue appears to be a duplicate of #18251

[heatherezell]

This issue appears to be a duplicate of #18251

So it is, thank you Max!