search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.96k stars 1.96k forks source link

Nomad is not loading the VAULT_TOKEN on a reload config triggered by a SIGHUP #24498

Open mario-fuentes opened 2 days ago

mario-fuentes commented 2 days ago

Nomad version

Nomad v1.8.4 BuildDate 2024-09-17T20:18:34Z Revision 22ab32e6cff66cf52f5e3f115b93de776bc09583

Operating system and Environment details

Ubuntu 20.04.1 LTS

Issue

When Nomad receive a SIGHUP to reload the configuration, it's not reloading the VAULT_TOKEN from the env var. We are using an vault-agent to get and refresh the Vault token used by the Nomad server, thus when Vault get a new token, a env file is renderer and a SIGHUP signal is sent to the Nomad process:

source /file/to/env && kill -HUP $NOMAD_PID

This approach is working fine in our production Nomad 1.6.8.

Note: our first attempt to upgrade to 1.7.7 failed by the same reason.

Reproduction steps

Start Nomad with Vault enabled but without set the VAULT_TOKEN env var (*), then send a SIGHUP to reload the config like:

$ export VAULT_TOKEN=PUT_A_VALID_TOKEN_HERE
$ kill -HUP $NOMAD_PID

(*) Prior to Nomad 1.7.x, the server startup process fail if the Vault integration is enabled and there is no a VAULT_TOKEN env var set, now the server start without the env var (maybe related to the issue)

Expected Result

Nomad reload the server configuration and upgrade the Vault token from the VAULT_TOKEN env var.

Actual Result

The new VAULT_TOKEN env var is ignored during the reload

pkazmierczak commented 6 hours ago

Hi @mario-fuentes, thanks for reporting this. In Nomad 1.7, we introduced Workload Identity based workflow for Consul and Vault integration. See here for the upgrade guide: https://developer.hashicorp.com/nomad/docs/upgrade/upgrade-specific#nomad-1-7-0

This means that while Nomad still supports setting the Vault token by env var or configuration variable, this workflow is deprecated from 1.7 and will be removed in upcoming Nomad version 1.10. We recommend running nomad setup vault -check to see if you can migrate your configuration to WI-based one.

All this being said, I will take a look at why 1.7 doesn't pick up ENV var automatically.

pkazmierczak commented 6 hours ago

@mario-fuentes, I reproduced it and indeed despite our move to WI-based tokens the legacy workflow should pick up token updates and it doesn't. I'll put it on our board and we'll work on fixing this.