Closed alekhrj closed 1 year ago
Hi there! Can you tell me more about your use case here, with using consul-replication for vault replication? Simply replicating the storage will not guarantee consistent results from Vault, as you've seen.
IIUC, the behaviour observed here actually counts as "Vault functioning as intended".
Vault expects and requires that it is the only thing that is writing to its storage, so by having consul-replicate write to storage that is also being used by a running Vault cluster, you create a situation where Vault will not function properly.
If you wanted to make this setup work, you would need to build a locking system so that consul-replicate, and Vault in the secondary datacenter, are never both running at the same time.
Even then, I'm not totally sure that consul-replicate would be able to preserve the ordering of operations well enough to guarantee it never places the destination Vault storage into a problematic state ... I'm thinking, what if it copies over storage entries encrypted by a new key, before the change to the keyring adding that new key?
I have some queries -
I am trying to understand the reason behind caching the nil data. one can be to prevent against DOS attack on non-existent key.
Also, I would like to understand what could possibly go wrong if we do not cache nil data as mentioned above - "you create a situation where Vault will not function properly."
"I'm thinking, what if it copies over storage entries encrypted by a new key, before the change to the keyring adding that new key?" -> are we talking about the rotation of root key scenario?
With your questions above, you've skipped past the really big important point to focus on some specific tiny parts of it, so I need to reiterate:
The Vault application is NOT written to support other software, such as consul-replicate, changing its data store underneath it whilst it is running.
You must not run consul-replicate whilst also running any Vault server in the secondary datacenter.
@maxb is correct here (thanks Max!) We cannot guarantee the results you're looking for if you change the storage. Caching, raft membership, and "eventual consistency" make this something that I don't believe we'll ever support. There are high-availability options available in Enterprise that can approximate what you're looking for (please see the page on Enterprise replication here ) but the way you're attempting to work around it will result in unexpected behavior at best and possible data corruption at worst. I'm going to close this issue now, but please feel free to open a new issue if you have further bug reports or enhancement requests. Thanks! :)
Although you're not using raft, apologies - the Integrated Storage versus Consul page makes it clear that Consul is all in-memory, which further complicates your use case here.
Describe the bug Setup : multi DC vault setup with consul as the storage backend and the consul data is being replicated using consul-replicate
Bug : vault caches all the responses (both correct and nil/error value) from the storage backend. If we try to read the value in the secondary DC before the replication, that data becomes unavailable even after successful replication
To Reproduce
Read a non-existent data in the secondary DC 1.1. Run
vault read transit/keys/non-existent-key-1
Create two keys (one with same name as in step 1.1) in the primary DC 2.1. Run
vault write -f transit/keys/non-existent-key-1
2.2. Runvault write -f transit/keys/non-existent-key-2
Read keys in the secondary cluster 3.1. Run
vault read transit/keys/non-existent-key-1
3.2. Runvault read transit/keys/non-existent-key-2
Force leader election in the secondary cluster. Run the following on the active node. 4.1. Run
vault operator step-down
Read keys in the secondary cluster 5.1. Run
vault read transit/keys/non-existent-key-1
5.2. Runvault read transit/keys/non-existent-key-2
Expected behavior Step 3.1 should give error - “Error : No value found at transit/keys/non-existent-key-1”
While steps 3.2, 5.1 and 5.2 should not give errors. Proving that there is no issue with the replication.
Similar behavior is observed in other backends as well such as kv and database.
Environment:
vault status
):cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" VERSION_CODENAME=stretch ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"