search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.81k stars 1.94k forks source link

Vault - Missing: vault.read(...) #22478

Closed fred-gb closed 3 months ago

fred-gb commented 3 months ago

Bonjour, 👋🏻

Nomad version

Nomad v1.8.0
BuildDate 2024-05-28T17:38:17Z
Revision 28b82e4b2259fae5a62e2ed47395334bea5a24c4

Operating system and Environment details

OS: Ubuntu 22.04 LTS
Vault: Vault v1.16.3 (e92d9a57018f43360e2e3717b3b6a7f650c88f4c), built 2024-05-29T14:28:42Z
Consul: Consul v1.18.2

mTLS / ACL enabled
Workload Identity enabled

Issue

Nomad is unable to start job, missing vault error message appears. But! Mongo example works.

Reproduction steps

Example mongo job, works:

job "mongo" {
  namespace = "default"
  node_pool = "all"

  group "db" {
    network {
      port "db" {
        static = 27017
      }
    }

    task "mongo" {
      driver = "docker"

      config {
        image = "mongo:7"
        ports = ["db"]
      }

      vault {}

      template {
        data        = <<EOF

MONGO_INITDB_ROOT_USERNAME=root

MONGO_INITDB_ROOT_PASSWORD={{with secret "apps/data/default/mongo/config"}}{{.Data.data.root_password}}{{end}}

EOF
        destination = "secrets/env"
        env         = true
      }
    }
  }
}

I write another secret:

vault kv put apps/other/mongo/config root_password=secret

And check:

vault kv get -mount="apps" "other/mongo/config"

======== Secret Path ========
apps/data/other/mongo/config

======= Metadata =======
Key                Value
---                -----
created_time       2024-06-01T14:07:37.536677988Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======== Data ========
Key              Value
---              -----
root_password    secret

I change template in Nomad job:

      template {
        data        = <<EOF

MONGO_INITDB_ROOT_USERNAME=root

MONGO_INITDB_ROOT_PASSWORD={{with secret "apps/data/other/mongo/config"}}{{.Data.data.root_password}}{{end}}

EOF
        destination = "secrets/env"
        env         = true
      }

Start job and!

[96014a17] Template: Missing: vault.read(apps/data/other/mongo/config)

Nomad Server logs (if appropriate)

Jun  1 16:13:37 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:37.034+0200 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="401.66µs"
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.710+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo/submission?version=35 duration="606.621µs"
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.778+0200 [DEBUG] http: request complete: method=POST path="/v1/jobs/parse?namespace=*" duration=2.448326ms
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.882+0200 [DEBUG] http: request complete: method=POST path=/v1/job/mongo duration=4.554984ms
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.883+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo?index=347 duration=9.016505888s
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.884+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo/evaluations?index=347 duration=8.992328746s
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.885+0200 [DEBUG] worker: dequeued evaluation: worker_id=46e7940d-9b21-811c-7ed0-2fee4004dd1b eval_id=425dc154-7a62-4fbf-cab6-f41ca2da04eb type=service namespace=default job_id=mongo node_id="" triggered_by=job-register
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.885+0200 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=425dc154-7a62-4fbf-cab6-f41ca2da04eb job_id=mongo namespace=default worker_id=46e7940d-9b21-811c-7ed0-2fee4004dd1b
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:   results=
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:   | Total changes: (place 1) (destructive 0) (inplace 0) (stop 0) (disconnect 0) (reconnect 0)
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:   | Created Deployment: "9295a227-4c52-02c0-55fc-3c2a6e2865ac"
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:   | Desired Changes for "db": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.893+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo/summary?index=349 duration=6.769579396s
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.894+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo/deployment?index=349 duration=6.765178038s
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.894+0200 [DEBUG] worker: submitted plan for evaluation: worker_id=46e7940d-9b21-811c-7ed0-2fee4004dd1b eval_id=425dc154-7a62-4fbf-cab6-f41ca2da04eb
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.895+0200 [DEBUG] worker.service_sched: setting eval status: eval_id=425dc154-7a62-4fbf-cab6-f41ca2da04eb job_id=mongo namespace=default worker_id=46e7940d-9b21-811c-7ed0-2fee4004dd1b status=complete
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.897+0200 [DEBUG] worker: updated evaluation: worker_id=46e7940d-9b21-811c-7ed0-2fee4004dd1b eval="<Eval \"425dc154-7a62-4fbf-cab6-f41ca2da04eb\" JobID: \"mongo\" Namespace: \"default\">"
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.898+0200 [DEBUG] worker: ack evaluation: worker_id=46e7940d-9b21-811c-7ed0-2fee4004dd1b eval_id=425dc154-7a62-4fbf-cab6-f41ca2da04eb type=service namespace=default job_id=mongo node_id="" triggered_by=job-register
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.898+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo/allocations?index=349 duration=6.760034057s
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.900+0200 [DEBUG] client: updated allocations: index=351 total=27 pulled=1 filtered=26
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.902+0200 [DEBUG] client: allocation updates: added=1 removed=0 updated=0 ignored=26
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.905+0200 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo type=Received msg="Task received by client" failed=false
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.905+0200 [DEBUG] client: allocation updates applied: added=1 removed=0 updated=0 ignored=26 errors=0
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.905+0200 [DEBUG] client.alloc_runner.widmgr: retrieving and renewing workload identities: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 num_identities=2
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.918+0200 [DEBUG] client.alloc_runner.widmgr: waiting to renew identities: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 num=2 wait=30m8.196125117s
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.935+0200 [DEBUG] client.alloc_runner.task_runner: lifecycle start condition has been met, proceeding: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.937+0200 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo type="Task Setup" msg="Building Task Directory" failed=false
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.940+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo path=/usr/bin/nomad args=["/usr/bin/nomad", "logmon"]
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.941+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo path=/usr/bin/nomad pid=23048
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.941+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo plugin=/usr/bin/nomad
Jun  1 16:13:38 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:38.994+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo?index=350 duration=1.069735ms
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.006+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo/evaluations?index=350 duration=2.00551ms
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.013+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo network=unix @module=logmon address=/tmp/plugin3881499240 timestamp="2024-06-01T16:13:39.013+0200"
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.013+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo version=2
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.017+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo path=/opt/nomad/alloc/3bb5cbf6-341b-8dd2-565f-37d68a89cfe5/alloc/logs/.mongo.stdout.fifo @module=logmon timestamp="2024-06-01T16:13:39.016+0200"
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.017+0200 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo @module=logmon path=/opt/nomad/alloc/3bb5cbf6-341b-8dd2-565f-37d68a89cfe5/alloc/logs/.mongo.stderr.fifo timestamp="2024-06-01T16:13:39.017+0200"
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.036+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo duration=2.032771ms
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.057+0200 [INFO]  agent: (runner) creating new runner (dry: false, once: false)
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.059+0200 [DEBUG] agent: (runner) final config: {"Consul":{"Address":"127.0.0.1:8501","Namespace":"","Auth":{"Enabled":false,"Username":""},"Retry":{"Attempts":0,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"/etc/ssl/hashistack/hashistack-ca.pem","CaCertBytes":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Token":"13b1f238-bd1b-336e-516a-83f9cec7c14b","TokenFile":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"WARN","FileLog":{"LogFilePath":"","LogRotateBytes":0,"LogRotateDuration":86400000000000,"LogRotateMaxFiles":0},"MaxStale":315360000000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"\nMONGO_INITDB_ROOT_USERNAME=root\n\nMONGO_INITDB_ROOT_PASSWORD={{with secret \"apps/data/other/mongo/config\"}}{{.Data.data.root_password}}{{end}}\n            \n","CreateDestDirs":true,"Destination":"/opt/nomad/alloc/3bb5cbf6-341b-8dd2-565f-37d68a89cfe5/mongo/secrets/env","ErrMissingKey":false,"ErrFatal":true,"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":["plugin","writeToFile"],"SandboxPath":"/opt/nomad/alloc/3bb5cbf6-341b-8dd2-565f-37d68a89cfe5/mongo","MapToEnvironmentVariable":""}],"TemplateErrFatal":null,"Vault":{"Address":"https://127.0.0.1:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":0,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"/etc/ssl/hashistack/hashistack-ca.pem","CaCertBytes":"","CaPath":"/etc/ssl/hashistack","Cert":"/etc/ssl/hashistack/dc1-server-vault.pem","Enabled":true,"Key":"/etc/ssl/hashistack/dc1-server-vault.key","ServerName":"","Verify":true},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"ClientUserAgent":null,"DefaultLeaseDuration":300000000000,"LeaseRenewalThreshold":0.9,"K8SAuthRoleName":"","K8SServiceAccountTokenPath":"/run/secrets/kubernetes.io/serviceaccount/token","K8SServiceAccountToken":"","K8SServiceMountPath":"kubernetes"},"Nomad":{"Address":"","Enabled":true,"Namespace":"default","SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"AuthUsername":"","AuthPassword":"","Transport":{"CustomDialer":{},"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"MaxConnsPerHost":0,"TLSHandshakeTimeout":10000000000},"Retry":{"Attempts":0,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true}},"Wait":{"Enabled":true,"Min":5000000000,"Max":240000000000},"Once":false,"ParseOnly":false,"BlockQueryWaitTime":300000000000,"ErrOnFailedLookup":false}
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.060+0200 [INFO]  agent: (runner) creating watcher
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.061+0200 [INFO]  agent: (runner) starting
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.062+0200 [DEBUG] agent: (runner) running initial templates
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.062+0200 [DEBUG] agent: (runner) initiating run
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.062+0200 [DEBUG] agent: (runner) checking template c303ccac719f71bf29531388264957fb
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.063+0200 [DEBUG] agent: (runner) missing data for 1 dependencies
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.064+0200 [DEBUG] agent: (runner) missing dependency: vault.read(apps/data/other/mongo/config)
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.064+0200 [DEBUG] agent: (runner) add used dependency vault.read(apps/data/other/mongo/config) to missing since isLeader but do not have a watcher
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.064+0200 [DEBUG] agent: (runner) was not watching 1 dependencies
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.065+0200 [DEBUG] agent: (watcher) adding vault.read(apps/data/other/mongo/config)
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.065+0200 [DEBUG] agent: (runner) diffing and updating dependencies
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.065+0200 [DEBUG] agent: (runner) watching 1 dependencies
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.105+0200 [WARN]  agent: (view) vault.read(apps/data/other/mongo/config): vault.read(apps/data/other/mongo/config): Error making API request.
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: URL: GET https://127.0.0.1:8200/v1/apps/data/other/mongo/config
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: Code: 403. Errors:
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: * 1 error occurred:
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: #011* permission denied
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:  (retry attempt 1 after "250ms")
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.106+0200 [ERROR] agent: (runner) sending server error back to caller
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.164+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo/allocations?index=351 duration=97.612819ms
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.165+0200 [DEBUG] client: updated allocations: index=353 total=27 pulled=0 filtered=27
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.167+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=27
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.167+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=27 errors=0
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.359+0200 [WARN]  agent: (view) vault.read(apps/data/other/mongo/config): vault.read(apps/data/other/mongo/config): Error making API request.
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: URL: GET https://127.0.0.1:8200/v1/apps/data/other/mongo/config
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: Code: 403. Errors:
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: * 1 error occurred:
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: #011* permission denied
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:  (retry attempt 2 after "500ms")
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.360+0200 [ERROR] agent: (runner) sending server error back to caller
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.360+0200 [DEBUG] agent: (runner) watching 1 dependencies
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.862+0200 [WARN]  agent: (view) vault.read(apps/data/other/mongo/config): vault.read(apps/data/other/mongo/config): Error making API request.
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: URL: GET https://127.0.0.1:8200/v1/apps/data/other/mongo/config
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: Code: 403. Errors:
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: * 1 error occurred:
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]: #011* permission denied
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:  (retry attempt 3 after "1s")
Jun  1 16:13:39 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:39.864+0200 [ERROR] agent: (runner) sending server error back to caller
Jun  1 16:13:40 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:40.868+0200 [WARN]  agent: (view) vault.read(apps/data/other/mongo/config): vault.read(apps/data/other/mongo/config): Error making API request.
Jun  1 16:13:40 pc09-quanticware-com nomad[12825]: URL: GET https://127.0.0.1:8200/v1/apps/data/other/mongo/config
Jun  1 16:13:40 pc09-quanticware-com nomad[12825]: Code: 403. Errors:
Jun  1 16:13:40 pc09-quanticware-com nomad[12825]: * 1 error occurred:
Jun  1 16:13:40 pc09-quanticware-com nomad[12825]: #011* permission denied
Jun  1 16:13:40 pc09-quanticware-com nomad[12825]:  (retry attempt 4 after "2s")
Jun  1 16:13:40 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:40.868+0200 [ERROR] agent: (runner) sending server error back to caller
Jun  1 16:13:40 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:40.868+0200 [DEBUG] agent: (runner) watching 1 dependencies
Jun  1 16:13:41 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:41.546+0200 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="421.846µs"
Jun  1 16:13:41 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:41.711+0200 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:48306
Jun  1 16:13:42 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:42.066+0200 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=3bb5cbf6-341b-8dd2-565f-37d68a89cfe5 task=mongo type=Template msg="Missing: vault.read(apps/data/other/mongo/config)" failed=false
Jun  1 16:13:42 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:42.243+0200 [DEBUG] client: updated allocations: index=354 total=27 pulled=0 filtered=27
Jun  1 16:13:42 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:42.245+0200 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=27
Jun  1 16:13:42 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:42.252+0200 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=27 errors=0
Jun  1 16:13:42 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:42.253+0200 [DEBUG] http: request complete: method=GET path=/v1/job/mongo/allocations?index=353 duration=1.178882741s
Jun  1 16:13:42 pc09-quanticware-com nomad[12825]:     2024-06-01T16:13:42.874+0200 [WARN]  agent: (view) vault.read(apps/data/other/mongo/config): vault.read(apps/data/other/mongo/config): Error making API request.

I tried with other jobs, but it doesn't work when I use Vault.

I don't understand.

When I installed a fresh hashisatck with token "future deprecated" and no workload identity method configured. It works well.

Need help !

This not a spam message 😅

fred-gb commented 3 months ago

Bonjour, 👋🏻

Sorry, misunderstood with documentation.

I finally understand Vault policy format. With this in the template: {{with secret "apps/data/default/mongo/config"}}

and policy

path "kv/data/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_namespace}}/{{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_job_id}}/*" {

so {{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_namespace}} = default ({{with secret "apps/data/default/mongo/config"}})

and {{identity.entity.aliases.AUTH_METHOD_ACCESSOR.metadata.nomad_job_id}}= mongo ({{with secret "apps/data/default/mongo/config"}})

Thanks!