search

hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
31.07k stars 4.2k forks source link

[ERROR] rollback: error rolling back: path= #22995

Open ErKaramSingh opened 1 year ago

ErKaramSingh commented 1 year ago

Describe the bug Observing below error messages post hashicorp vault update Update: v1.9.3 -> 1.12.3 -> 1.14.0 Helm chart: v0.20.1 -> v0.25.0 GKE version: 1.26.5-gke.1200

We are not sure if the below problem persists in earlier versions of hashicorp vault but we observed it first on hashicorp vault v1.9.3 and also when we were trying to update to v1.14.0 recently. And immediately post vault unseal the below errors start appearing and they recur every 1 hour.

[ERROR] rollback: error rolling back: path=data-protection-keys/customer_id/
error=
| 326608 errors occurred:
|
[resource.labels.containerName: vault] | \t* failed to read value for "logical/20f9cb23-edbb-e9a1-f111-47b7bae86e4a/policy/dmF1bHQ6djE6dDRaRWgyVFZKRy9YdGZPZFFUWmM4OVpTTkpjaUg3RXppYzFtakRxUzZjR25rT09sZkhXcWV1Z2RNN1V1c0QzZkVKUlR1S3cx": context deadline exceeded
[resource.labels.containerName: vault] | \t* failed to read value for "logical/20f9cb23-edbb-e9a1-f111-47b7bae86e4a/policy/dmF1bHQ6djE6dDRjRm1ZcWtPWGRxY3h5ZU1WeklRcUxEanVzNkY3ZXFqY3RrVHdLWDMzZm5tQWZzeXpVeGpaQU5IcmlPOTJQNDZwV3Z0ZEc5": context deadline exceeded
[resource.labels.containerName: vault] | \t* failed to read value for "logical/20f9cb23-edbb-e9a1-f111-47b7bae86e4a/policy/dmF1bHQ6djE6dDRmQmFsRlRGSGQ4OHVqVWkyaGJmZ0s1bE9CRmdRU1NRLzZNR2NBaTQ1aTQ4RENCYjNFbDYrSjAxdEcxMDdkQWkzZU1yTWRy": context deadline exceeded

To Reproduce

This seems to be problem related to specific secret setting which is unable to read value from the specific folder in GCS bucket see above logs. Not sure if i need to make any changes for below line of code:

vault secrets enable -path="data-protection-keys/customer_id" -force-no-cache transit

Steps to reproduce the behavior:

  1. Run vault secrets list -detailed

    vault secrets list -detailed
Path                                           Plugin       Accessor              Default TTL    Max TTL    Force No Cache    Replication    Seal Wrap    External Entropy Access    Options           Description                                                UUID                                    Version    Running Version          Running SHA256    Deprecation Status
----                                           ------       --------              -----------    -------    --------------    -----------    ---------    -----------------------    -------           -----------                                                ----                                    -------    ---------------          --------------    ------------------
backup/customer-keys/                          transit      transit_7b4e021a      system         system     false             replicated     false        false                      map[]             n/a                                                        db9bb5b3-d3ce-0236-d1e8-5a4efe8e6c3b    n/a        v1.12.3+builtin.vault    n/a               supported

  2. Run vault secrets enable -path="data-protection-keys/customer_id_test" -force-no-cache transit vault secrets enable -path="data-protection-keys/customer_id_test" -force-no-cache transit Success! Enabled the transit secrets engine at: data-protection-keys/customer_id_test/

    vault secrets list -detailed
Path                                           Plugin       Accessor              Default TTL    Max TTL    Force No Cache    Replication    Seal Wrap    External Entropy Access    Options           Description                                                UUID                                    Version    Running Version          Running SHA256    Deprecation Status
----                                           ------       --------              -----------    -------    --------------    -----------    ---------    -----------------------    -------           -----------                                                ----                                    -------    ---------------          --------------    ------------------
data-protection-keys/customer_id_test/         transit      transit_47087131      system         system     true              replicated     false        false                      map[]             n/a                                                        1ecf82d0-714f-cc58-67d7-4838b15d0b4a    n/a        v1.12.3+builtin.vault    n/a               supported

  3. Run 'vault secrets move data-protection-keys/customer_id_test backup/data-protection-keys/customer_id_test'

    vault secrets move data-protection-keys/customer_id_test  backup/data-protection-keys/customer_id_test
Started moving secrets engine data-protection-keys/customer_id_test/ to backup/data-protection-keys/customer_id_test/, with migration ID 3392d0b3-498a-aec7-1062-1c507dc3784a
Waiting for terminal status in migration of secrets engine data-protection-keys/customer_id_test/ to backup/data-protection-keys/customer_id_test/, with migration ID 3392d0b3-498a-aec7-1062-1c507dc3784a
Success! Finished moving secrets engine data-protection-keys/customer_id_test/ to backup/data-protection-keys/customer_id_test/, with migration ID 3392d0b3-498a-aec7-1062-1c507dc3784a

  4. Run 'vault secrets disable backup/data-protection-keys/customer_id_test/'

    vault secrets disable backup/data-protection-keys/customer_id_test/
Success! Disabled the secrets engine (if it existed) at: backup/data-protection-keys/customer_id_test/

Expected behavior force-no-cache is enabled for dev environment and observe "force-no-cache:true" persists. we expected the same for the above infra environment.

Environment: infra

vault secrets list -detailed
Path                                           Plugin       Accessor              Default TTL    Max TTL    Force No Cache    Replication    Seal Wrap    External Entropy Access    Options           Description                                                UUID                                    Version    Running Version          Running SHA256    Deprecation Status
----                                           ------       --------              -----------    -------    --------------    -----------    ---------    -----------------------    -------           -----------                                                ----                                    -------    ---------------          --------------    ------------------
backup/customer-keys/                          transit      transit_7b4e021a      system         system     false             replicated     false        false                      map[]             n/a                                                        db9bb5b3-d3ce-0236-d1e8-5a4efe8e6c3b    n/a        v1.12.3+builtin.vault    n/a               supported

Environment: dev

#vault secrets list -detailed
Path                                           Plugin       Accessor              Default TTL    Max TTL    Force No Cache    Replication    Seal Wrap    External Entropy Access    Options                     Description                                                UUID                                    Version    Running Version          Running SHA256    Deprecation Status
----                                           ------       --------              -----------    -------    --------------    -----------    ---------    -----------------------    -------                     -----------                                                ----                                    -------    ---------------          --------------    ------------------
backup/customer-keys/                          transit      transit_6178b21a      system         system     false             replicated     false        false                      map[force-no-cache:true]    n/a                                                        bfdb84aa-58af-afbd-c320-e647b1428ab0    n/a        v1.12.3+builtin.vault    n/a               supported

Environment:

* Vault Server Version (retrieve with `vault status`): 
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.12.3
Build Date      2023-02-02T09:07:27Z
Storage Type    gcs
Cluster Name    vault-cluster-911cd146
Cluster ID      9e03802f-f4e0-a001-78f6-3b9ea364581c
HA Enabled      true
HA Cluster      https://infraqa-vault-0.infraqa-vault-internal:8201
HA Mode         active
Active Since    2023-09-12T09:23:48.404038572Z

Additional context Add any other context about the problem here.

In past we had some memory leak issue where in we were suggested to enable force-no-cache for the secret path. See link : https://github.com/hashicorp/vault/issues/5746

The same can't be enabled in hashicorp version v1.9.3 and v1.12.3 as well. Is it a bug in hashicorp vault or how can we activate this functionality.

craftey commented 1 year ago

The issue can be reproduced like described here: https://github.com/hashicorp/vault/issues/23566