search

hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
31.34k stars 4.23k forks source link

API documentation is misleading, or API implementation is misleading – drift between doc and behaviour. #17953

Open aamkye opened 2 years ago

aamkye commented 2 years ago

Describe the bug Due to cloud-agnostic approach, I'm building auto-unseal application. Despite my need, I'm using vault API REST calls to the vault servers. The problem is that API sometimes lies. For example, UNSEAL API /sys/unseal with sample-response says that during unsealing progress shows how many keys have been provided, and then resets to 0 while vault is unsealed. And at the same time, sealed field should point to the false. Which is not true (example below):

What is more I've added retry logic due to responses from API (can be observed in 2nd app log) - without repeat logic, unsealing fails.

To Reproduce Steps to reproduce the behaviour:

  1. Run /sys/init on leader
  2. Run 5x /sys/unseal on leader
  3. Run /sys/storage/raft/join on 2x follower
  4. Run 5x /sys/unseal on 2x follower
  5. See error in API responses related to progress and sealed

Log from application with sleeps between actions ~2000ms

{"@message":"Received POST /init","@timestamp":"08:52:24.162038Z"}
{"@message":"Vault on 10.42.0.161: Initialized - true","@timestamp":"08:52:31.569892Z"}
{"@message":"Vault on 10.42.0.161 (Initialized: true, Sealed: true): Unseal progress: (provided 1 of 5; max 5).","@timestamp":"08:52:31.577540Z"}
{"@message":"Vault on 10.42.0.161 (Initialized: true, Sealed: true): Unseal progress: (provided 2 of 5; max 5).","@timestamp":"08:52:33.583486Z"}
{"@message":"Vault on 10.42.0.161 (Initialized: true, Sealed: true): Unseal progress: (provided 3 of 5; max 5).","@timestamp":"08:52:35.603711Z"}
{"@message":"Vault on 10.42.0.161 (Initialized: true, Sealed: true): Unseal progress: (provided 4 of 5; max 5).","@timestamp":"08:52:37.610396Z"}
{"@message":"Vault on 10.42.0.161 (Initialized: true, Sealed: false): Unseal progress: (provided 0 of 5; max 5).","@timestamp":"08:52:39.849747Z"}
{"@message":"Vault on 10.42.0.161 is unsealed.","@timestamp":"08:52:39.850750Z"}
{"@message":"Vault check on 10.42.0.162 (Initialized: true, Sealed: true)","@timestamp":"08:52:46.854904Z"}
{"@message":"Vault on 10.42.0.162: Joined to 10.42.0.161 cluster - true","@timestamp":"08:52:46.877172Z"}
{"@message":"Vault check on 10.42.0.163 (Initialized: true, Sealed: true)","@timestamp":"08:52:48.930440Z"}
{"@message":"Vault on 10.42.0.163: Joined to 10.42.0.161 cluster - true","@timestamp":"08:52:49.019859Z"}
{"@message":"Vault on 10.42.0.162 (Initialized: true, Sealed: true): Unseal progress: (provided 1 of 5; max 5).","@timestamp":"08:52:51.028718Z"}
{"@message":"Vault on 10.42.0.162 (Initialized: true, Sealed: true): Unseal progress: (provided 2 of 5; max 5).","@timestamp":"08:52:53.038433Z"}
{"@message":"Vault on 10.42.0.162 (Initialized: true, Sealed: true): Unseal progress: (provided 3 of 5; max 5).","@timestamp":"08:52:55.048318Z"}
{"@message":"Vault on 10.42.0.162 (Initialized: true, Sealed: true): Unseal progress: (provided 4 of 5; max 5).","@timestamp":"08:52:57.058283Z"}
{"@message":"Vault on 10.42.0.162 (Initialized: true, Sealed: true): Unseal progress: (provided 0 of 5; max 5).","@timestamp":"08:53:00.156612Z"}
{"@message":"Vault on 10.42.0.162 (Initialized: true, Sealed: false): Unseal progress: (provided 0 of 5; max 5).","@timestamp":"08:53:02.168162Z"}
{"@message":"Vault on 10.42.0.162 is unsealed.","@timestamp":"08:53:02.168289Z"}
{"@message":"Vault on 10.42.0.163 (Initialized: true, Sealed: true): Unseal progress: (provided 1 of 5; max 5).","@timestamp":"08:53:04.185663Z"}
{"@message":"Vault on 10.42.0.163 (Initialized: true, Sealed: true): Unseal progress: (provided 2 of 5; max 5).","@timestamp":"08:53:06.192819Z"}
{"@message":"Vault on 10.42.0.163 (Initialized: true, Sealed: true): Unseal progress: (provided 3 of 5; max 5).","@timestamp":"08:53:08.213168Z"}
{"@message":"Vault on 10.42.0.163 (Initialized: true, Sealed: true): Unseal progress: (provided 4 of 5; max 5).","@timestamp":"08:53:10.227792Z"}
{"@message":"Vault on 10.42.0.163 (Initialized: true, Sealed: true): Unseal progress: (provided 0 of 5; max 5).","@timestamp":"08:53:14.279705Z"}
{"@message":"Vault on 10.42.0.163 (Initialized: true, Sealed: false): Unseal progress: (provided 0 of 5; max 5).","@timestamp":"08:53:16.284129Z"}
{"@message":"Vault on 10.42.0.163 is unsealed.","@timestamp":"08:53:16.284208Z"}
{"@message":"Vault initialization finished.","@timestamp":"08:53:18.285074Z"

Log from application with sleeps between actions ~200ms

{"@message":"Received POST /init","@timestamp":"2022-11-16T09:30:12.808061Z"}
{"@message":"Vault on 10.42.0.170 (Initialized: true, Sealed: true): Unseal progress: (provided 1 of 5; max 5).","@timestamp":"2022-11-16T09:30:22.798053Z"}
{"@message":"Vault on 10.42.0.170 (Initialized: true, Sealed: true): Unseal progress: (provided 2 of 5; max 5).","@timestamp":"2022-11-16T09:30:23.001344Z"}
{"@message":"Vault on 10.42.0.170 (Initialized: true, Sealed: true): Unseal progress: (provided 3 of 5; max 5).","@timestamp":"2022-11-16T09:30:23.205563Z"}
{"@message":"Vault on 10.42.0.170 (Initialized: true, Sealed: true): Unseal progress: (provided 4 of 5; max 5).","@timestamp":"2022-11-16T09:30:23.429495Z"}
{"@message":"Vault on 10.42.0.170 (Initialized: true, Sealed: false): Unseal progress: (provided 0 of 5; max 5).","@timestamp":"2022-11-16T09:30:23.841320Z"}
{"@message":"Vault on 10.42.0.170 is unsealed.","@timestamp":"2022-11-16T09:30:23.842428Z"}
{"@message":"Vault check on 10.42.0.171 (Initialized: true, Sealed: true)","@timestamp":"2022-11-16T09:30:29.080040Z"}
{"@message":"Vault on 10.42.0.171: Joined to 10.42.0.170 cluster - true","@timestamp":"2022-11-16T09:30:29.131084Z"}
{"@message":"Vault check on 10.42.0.172 (Initialized: true, Sealed: true)","@timestamp":"2022-11-16T09:30:29.335673Z"}
{"@message":"Vault on 10.42.0.172: Joined to 10.42.0.170 cluster - true","@timestamp":"2022-11-16T09:30:29.377843Z"}
{"@message":"Vault on 10.42.0.171 (Initialized: true, Sealed: true): Unseal progress: (provided 1 of 5; max 5).","@timestamp":"2022-11-16T09:30:29.592834Z"}
{"@message":"Vault on 10.42.0.171 (Initialized: true, Sealed: true): Unseal progress: (provided 2 of 5; max 5).","@timestamp":"2022-11-16T09:30:29.798797Z"}
{"@message":"Vault on 10.42.0.171 (Initialized: true, Sealed: true): Unseal progress: (provided 3 of 5; max 5).","@timestamp":"2022-11-16T09:30:30.005341Z"}
{"@message":"Vault on 10.42.0.171 (Initialized: true, Sealed: true): Unseal progress: (provided 4 of 5; max 5).","@timestamp":"2022-11-16T09:30:30.212818Z"}
{"@message":"Vault on 10.42.0.171 (Initialized: true, Sealed: true): Unseal progress: (provided 0 of 5; max 5).","@timestamp":"2022-11-16T09:30:30.456733Z"}
{"@message":"Vault on 10.42.0.171 (Initialized: true, Sealed: true): Unseal progress: (provided 1 of 5; max 5).","@timestamp":"2022-11-16T09:30:30.667591Z"}
{"@message":"Vault on 10.42.0.171 (Initialized: true, Sealed: true): Unseal progress: (provided 2 of 5; max 5).","@timestamp":"2022-11-16T09:30:30.876522Z"}
{"@message":"Vault on 10.42.0.171 (Initialized: true, Sealed: true): Unseal progress: (provided 3 of 5; max 5).","@timestamp":"2022-11-16T09:30:31.080955Z"}
{"@message":"Vault on 10.42.0.171 (Initialized: true, Sealed: true): Unseal progress: (provided 4 of 5; max 5).","@timestamp":"2022-11-16T09:30:31.285931Z"}
{"@message":"Vault on 10.42.0.171 (Initialized: true, Sealed: false): Unseal progress: (provided 4 of 5; max 5).","@timestamp":"2022-11-16T09:30:31.493470Z"}
{"@message":"Vault on 10.42.0.171 is unsealed.","@timestamp":"2022-11-16T09:30:31.493591Z"}
{"@message":"Vault on 10.42.0.172 (Initialized: true, Sealed: true): Unseal progress: (provided 1 of 5; max 5).","@timestamp":"2022-11-16T09:30:31.703944Z"}
{"@message":"Vault on 10.42.0.172 (Initialized: true, Sealed: true): Unseal progress: (provided 2 of 5; max 5).","@timestamp":"2022-11-16T09:30:31.913557Z"}
{"@message":"Vault on 10.42.0.172 (Initialized: true, Sealed: true): Unseal progress: (provided 3 of 5; max 5).","@timestamp":"2022-11-16T09:30:32.120615Z"}
{"@message":"Vault on 10.42.0.172 (Initialized: true, Sealed: true): Unseal progress: (provided 4 of 5; max 5).","@timestamp":"2022-11-16T09:30:32.329357Z"}
{"@message":"Vault on 10.42.0.172 (Initialized: true, Sealed: true): Unseal progress: (provided 0 of 5; max 5).","@timestamp":"2022-11-16T09:30:33.117454Z"}
{"@message":"Vault on 10.42.0.172 (Initialized: true, Sealed: true): Unseal progress: (provided 1 of 5; max 5).","@timestamp":"2022-11-16T09:30:33.321064Z"}
{"@message":"Vault on 10.42.0.172 (Initialized: true, Sealed: true): Unseal progress: (provided 2 of 5; max 5).","@timestamp":"2022-11-16T09:30:33.525388Z"}
{"@message":"Vault on 10.42.0.172 (Initialized: true, Sealed: true): Unseal progress: (provided 3 of 5; max 5).","@timestamp":"2022-11-16T09:30:33.731455Z"}
{"@message":"Vault on 10.42.0.172 (Initialized: true, Sealed: true): Unseal progress: (provided 4 of 5; max 5).","@timestamp":"2022-11-16T09:30:33.936462Z"}
{"@message":"Vault on 10.42.0.172 (Initialized: true, Sealed: false): Unseal progress: (provided 4 of 5; max 5).","@timestamp":"2022-11-16T09:30:34.144861Z"}
{"@message":"Vault on 10.42.0.172 is unsealed.","@timestamp":"2022-11-16T09:30:34.144921Z"}
{"@message":"Vault initialization finished.","@timestamp":"2022-11-16T09:30:34.348171Z"}

Expected behaviour Every unseal operation returns “true” state according to vault state.

Environment:

Vault server configuration file(s):

listener "tcp" {
  tls_disable = 0
  address = "[::]:8200"
  cluster_address = "[::]:8201"
  tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
  tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
  tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
}

storage "raft" {
  path = "/vault/data"

  retry_join {
    leader_api_addr = "https://vault-0.vault-internal:8200"
    leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
    leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
    leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
  }

  retry_join {
    leader_api_addr = "https://vault-1.vault-internal:8200"
    leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
    leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
    leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
  }

  retry_join {
    leader_api_addr = "https://vault-2.vault-internal:8200"
    leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
    leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
    leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
  }

  autopilot {
    last_contact_threshold = "200ms"
    last_contact_failure_threshold = "10m"
    max_trailing_logs = 250000
    min_quorum = 2
    server_stabilization_time = "10s"
  }
}

disable_mlock = true
service_registration "kubernetes" {}

Additional context Please feel free to contact me for live demo of this problem and discuss the details.

aamkye commented 2 years ago

Also, my assumption is that this request to the vault is a non-blocking, so the vault cannot keep up with the state update and returns old state. Please correct me if I'm mistaken.

oferchen commented 1 year ago

I have the same use case.