search

hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
31.18k stars 4.21k forks source link

token_file authentication fails after setting up according to documentation #28011

Closed HomeOfTheWizard closed 3 months ago

HomeOfTheWizard commented 3 months ago

Describe the bug Hi, I am trying to run an agent container fetching kv secrets from a vault server run in dev mode in another docker container. The token_file is not accepted after setting up according to this documentation https://developer.hashicorp.com/vault/docs/agent-and-proxy/autoauth/methods/token_file https://developer.hashicorp.com/vault/tutorials/vault-agent/agent-quick-start Can you please help ?

To Reproduce Project can be found here: https://github.com/HomeOfTheWizard/vault-mvnd-benchmark

Steps to reproduce the behavior:

  1. Run vault server via the script launch-vault.sh
  2. Checking logs with docker compose -f docker/vault/docker-compose-vault.yaml logs 
    ozgun@ozgun-GL553VD:~/IdeaProjects/vault-mvnd-benchmark$ docker compose -f docker/vault/docker-compose-vault.yaml logs
vault  | ==> Vault server configuration:
vault  | 
vault  | Administrative Namespace: 
vault  |              Api Address: http://0.0.0.0:8200
vault  |                      Cgo: disabled
vault  |          Cluster Address: https://0.0.0.0:8201
vault  |    Environment Variables: GOTRACEBACK, HOME, HOSTNAME, NAME, PATH, PWD, SHLVL, VAULT_ADDR, VAULT_DEV_ROOT_TOKEN_ID, VAULT_LOCAL_CONFIG, VERSION
vault  |               Go Version: go1.21.9
vault  |               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", disable_request_limiter: "false", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault  |                Log Level: 
vault  |                    Mlock: supported: true, enabled: false
vault  |            Recovery Mode: false
vault  |                  Storage: inmem
vault  |                  Version: Vault v1.16.2, built 2024-04-22T16:25:54Z
vault  |              Version Sha: c6e4c2d4dc3b0d57791881b087c026e2f75a87cb
vault  | 
vault  | ==> Vault server started! Log data will stream in below:
vault  | 
vault  | 2024-08-07T12:58:52.887Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
vault  | 2024-08-07T12:58:52.887Z [INFO]  incrementing seal generation: generation=1
vault  | 2024-08-07T12:58:52.887Z [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
vault  | 2024-08-07T12:58:52.888Z [INFO]  core: Initializing version history cache for core
vault  | 2024-08-07T12:58:52.888Z [INFO]  events: Starting event system
vault  | 2024-08-07T12:58:52.888Z [INFO]  core: security barrier not initialized
vault  | 2024-08-07T12:58:52.889Z [INFO]  core: security barrier initialized: stored=1 shares=1 threshold=1
vault  | 2024-08-07T12:58:52.889Z [INFO]  core: post-unseal setup starting
vault  | 2024-08-07T12:58:52.897Z [INFO]  core: loaded wrapping token key
vault  | 2024-08-07T12:58:52.897Z [INFO]  core: successfully setup plugin runtime catalog
vault  | 2024-08-07T12:58:52.897Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
vault  | 2024-08-07T12:58:52.897Z [INFO]  core: no mounts; adding default mount table
vault  | 2024-08-07T12:58:52.898Z [INFO]  core: successfully mounted: type=cubbyhole version="v1.16.2+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
vault  | 2024-08-07T12:58:52.899Z [INFO]  core: successfully mounted: type=system version="v1.16.2+builtin.vault" path=sys/ namespace="ID: root. Path: "
vault  | 2024-08-07T12:58:52.899Z [INFO]  core: successfully mounted: type=identity version="v1.16.2+builtin.vault" path=identity/ namespace="ID: root. Path: "
vault  | 2024-08-07T12:58:52.901Z [INFO]  core: successfully mounted: type=token version="v1.16.2+builtin.vault" path=token/ namespace="ID: root. Path: "
vault  | 2024-08-07T12:58:52.901Z [INFO]  rollback: Starting the rollback manager with 256 workers
vault  | 2024-08-07T12:58:52.901Z [INFO]  rollback: starting rollback manager
vault  | 2024-08-07T12:58:52.902Z [INFO]  core: restoring leases
vault  | 2024-08-07T12:58:52.902Z [INFO]  expiration: lease restore complete
vault  | 2024-08-07T12:58:52.903Z [INFO]  identity: entities restored
vault  | 2024-08-07T12:58:52.903Z [INFO]  identity: groups restored
vault  | 2024-08-07T12:58:52.903Z [INFO]  core: Recorded vault version: vault version=1.16.2 upgrade time="2024-08-07 12:58:52.903164834 +0000 UTC" build date=2024-04-22T16:25:54Z
vault  | 2024-08-07T12:58:52.903Z [INFO]  core: post-unseal setup complete
vault  | 2024-08-07T12:58:52.903Z [INFO]  core: root token generated
vault  | 2024-08-07T12:58:52.903Z [INFO]  core: pre-seal teardown starting
vault  | 2024-08-07T12:58:52.903Z [INFO]  rollback: stopping rollback manager
vault  | 2024-08-07T12:58:52.904Z [INFO]  core: pre-seal teardown complete
vault  | 2024-08-07T12:58:52.904Z [INFO]  core.cluster-listener.tcp: starting listener: listener_address=0.0.0.0:8201
vault  | 2024-08-07T12:58:52.904Z [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
vault  | 2024-08-07T12:58:52.904Z [INFO]  core: post-unseal setup starting
vault  | 2024-08-07T12:58:52.904Z [INFO]  core: loaded wrapping token key
vault  | 2024-08-07T12:58:52.904Z [INFO]  core: successfully setup plugin runtime catalog
vault  | 2024-08-07T12:58:52.904Z [INFO]  core: successfully setup plugin catalog: plugin-directory=""
vault  | 2024-08-07T12:58:52.905Z [INFO]  core: successfully mounted: type=system version="v1.16.2+builtin.vault" path=sys/ namespace="ID: root. Path: "
vault  | 2024-08-07T12:58:52.905Z [INFO]  core: successfully mounted: type=identity version="v1.16.2+builtin.vault" path=identity/ namespace="ID: root. Path: "
vault  | 2024-08-07T12:58:52.905Z [INFO]  core: successfully mounted: type=cubbyhole version="v1.16.2+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
vault  | 2024-08-07T12:58:52.906Z [INFO]  core: successfully mounted: type=token version="v1.16.2+builtin.vault" path=token/ namespace="ID: root. Path: "
vault  | 2024-08-07T12:58:52.906Z [INFO]  rollback: Starting the rollback manager with 256 workers
vault  | 2024-08-07T12:58:52.906Z [INFO]  rollback: starting rollback manager
vault  | 2024-08-07T12:58:52.907Z [INFO]  core: restoring leases
vault  | 2024-08-07T12:58:52.907Z [INFO]  expiration: lease restore complete
vault  | 2024-08-07T12:58:52.907Z [INFO]  identity: entities restored
vault  | 2024-08-07T12:58:52.907Z [INFO]  identity: groups restored
vault  | 2024-08-07T12:58:52.907Z [INFO]  core: post-unseal setup complete
vault  | 2024-08-07T12:58:52.907Z [INFO]  core: vault is unsealed
vault  | 2024-08-07T12:58:52.909Z [INFO]  expiration: revoked lease: lease_id=auth/token/root/h579136c44fc94c1004ac5a7024998121a3fdbbb4135806a61c097ff1a2cde840
vault  | 2024-08-07T12:58:52.912Z [INFO]  core: successful mount: namespace="" path=secret/ type=kv version="v0.17.0+builtin"
vault  | WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
vault  | and starts unsealed with a single unseal key. The root token is already
vault  | authenticated to the CLI, so you can immediately begin using Vault.
vault  | 
vault  | You may need to set the following environment variables:
vault  | 
vault  |     $ export VAULT_ADDR='http://0.0.0.0:8200'
vault  | 
vault  | The unseal key and root token are displayed below in case you want to
vault  | seal/unseal the Vault or re-authenticate.
vault  | 
vault  | Unseal Key: 2rQliNF8TjLQW2YJvl7CnVwHj5mXN/Wkmt+13EXoCiQ=
vault  | Root Token: 00000000-0000-0000-0000-000000000000
vault  | 
vault  | Development mode should NOT be used in production installations!
  3. Run agent via the script launch-agent.sh
  4. See error by checking the agent container logs 
    ozgun@ozgun-GL553VD:~/IdeaProjects/vault-mvnd-benchmark$ docker compose -f docker/vault/docker-compose-vault-agent.yaml logs
vault-agent  | ==> Vault Agent started! Log data will stream in below:
vault-agent  | 
vault-agent  | ==> Vault Agent configuration:
vault-agent  | 
vault-agent  |            Api Address 1: http://bufconn
vault-agent  |                      Cgo: disabled
vault-agent  |                Log Level: debug
vault-agent  |                  Version: Vault v1.16.2, built 2024-04-22T16:25:54Z
vault-agent  |              Version Sha: c6e4c2d4dc3b0d57791881b087c026e2f75a87cb
vault-agent  | 
vault-agent  | 2024-08-07T12:24:09.214Z [INFO]  agent.sink.file: creating file sink
vault-agent  | 2024-08-07T12:24:09.214Z [INFO]  agent.sink.file: file sink configured: path=/vault/token/vault-token-via-agent mode=-rw-r-----
vault-agent  | 2024-08-07T12:24:09.215Z [DEBUG] agent: would have sent systemd notification (systemd not present): notification=READY=1
vault-agent  | 2024-08-07T12:24:09.215Z [INFO]  agent.exec.server: starting exec server
vault-agent  | 2024-08-07T12:24:09.215Z [INFO]  agent.exec.server: no env templates or exec config, exiting
vault-agent  | 2024-08-07T12:24:09.215Z [INFO]  agent.sink.server: starting sink server
vault-agent  | 2024-08-07T12:24:09.215Z [INFO]  agent.auth.handler: starting auth handler
vault-agent  | 2024-08-07T12:24:09.215Z [INFO]  agent.auth.handler: authenticating
vault-agent  | 2024-08-07T12:24:09.215Z [INFO]  agent.template.server: starting template server
vault-agent  | 2024-08-07T12:24:09.215Z [INFO]  agent: (runner) creating new runner (dry: false, once: false)
vault-agent  | 2024-08-07T12:24:09.215Z [ERROR] agent.auth.handler: token file validation failed, token may be invalid: backoff=950ms
vault-agent  | 2024-08-07T12:24:09.215Z [DEBUG] agent: (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","TokenFile":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","FileLog":{"LogFilePath":"","LogRotateBytes":0,"LogRotateDuration":86400000000000,"LogRotateMaxFiles":0},"MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"  {{- with secret \"kv/data/fruit-basket\" }}\nPRODUCER_NAME: {{ Data.data.producer_name }}\nPRODUCER_PSW: {{ Data.data.producer_password }}\nPRODUCER_FRUIT: {{ Data.data.producer_fruit }}\n  {{ end }}\n  ","CreateDestDirs":true,"Destination":"/vault/secrets/application.yaml","ErrMissingKey":false,"ErrFatal":true,"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionDenylist":[],"SandboxPath":"","MapToEnvironmentVariable":""}],"TemplateErrFatal":null,"Vault":{"Address":"http://0.0.0.0:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":10,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"ClientUserAgent":null,"DefaultLeaseDuration":300000000000,"LeaseRenewalThreshold":0.9,"K8SAuthRoleName":"","K8SServiceAccountTokenPath":"/run/secrets/kubernetes.io/serviceaccount/token","K8SServiceAccountToken":"","K8SServiceMountPath":"kubernetes"},"Nomad":{"Address":"","Enabled":false,"Namespace":"","SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"AuthUsername":"","AuthPassword":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true}},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"ParseOnly":false,"BlockQueryWaitTime":60000000000,"ErrOnFailedLookup":false}
vault-agent  | 2024-08-07T12:24:09.215Z [INFO]  agent: (runner) creating watcher
vault-agent  | 2024-08-07T12:24:10.174Z [INFO]  agent.auth.handler: authenticating
vault-agent  | 2024-08-07T12:24:10.174Z [ERROR] agent.auth.handler: token file validation failed, token may be invalid: backoff=950ms
vault-agent  | 2024-08-07T12:24:11.827Z [INFO]  agent.auth.handler: authenticating
vault-agent  | 2024-08-07T12:24:11.828Z [ERROR] agent.auth.handler: token file validation failed, token may be invalid: backoff=1.65s
vault-agent  | 2024-08-07T12:24:14.877Z [INFO]  agent.auth.handler: authenticating
vault-agent  | 2024-08-07T12:24:14.878Z [ERROR] agent.auth.handler: token file validation failed, token may be invalid: backoff=3.04s
vault-agent  | 2024-08-07T12:24:19.933Z [INFO]  agent.auth.handler: authenticating
vault-agent  | 2024-08-07T12:24:19.934Z [ERROR] agent.auth.handler: token file validation failed, token may be invalid: backoff=5.05s

However we can see the token file with the correct token by connecting to the container

ozgun@ozgun-GL553VD:~/IdeaProjects/vault-mvnd-benchmark$ docker exec -it vault-agent sh 
/ # cat /vault/token/.vault-token 
00000000-0000-0000-0000-000000000000/ #

Expected behavior Authenticate to the server started in dev mode with the provided root token. And then fetch the secrets using the template.

Environment:

Vault agent configuration file(s):

pid_file = "/vault/.pidfile"

vault {
  address = "$VAULT_ADDR"
  tls_skip_verify = true
}

auto_auth {
  method {
    type = "token_file"
    config = {
      token_file_path = "/vault/token/.vault-token"
    }
  }
  sink "file" {
    config = {
      path = "/vault/token/vault-token-via-agent"
    }
  }
}

template {
  destination = "/vault/secrets/application.yaml"
  contents = <<EOT
  {{- with secret "kv/data/fruit-basket" }}
PRODUCER_NAME: {{ Data.data.producer_name }}
PRODUCER_PSW: {{ Data.data.producer_password }}
PRODUCER_FRUIT: {{ Data.data.producer_fruit }}
  {{ end }}
  EOT
}
HomeOfTheWizard commented 3 months ago

it was the name of the service that must be used between the agent and server containers, instead of ip. the "$VAULT_ADDR" was pointing to the ip instead of compose service name. mea culpa