Nomad uniqueness of consul agent is confusing

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

https://www.nomadproject.io/

Other

14.89k stars 1.95k forks source link

Nomad uniqueness of consul agent is confusing #7639

Open notnoop opened 4 years ago

notnoop commented 4 years ago

Nomad agents are expected to have unique mapping to consul agents: No two nomad agents should communicate to the same consul agent.

Though documented[1], the failure mode is very confusing and is hard to debug. It appears as nomad job services get removed unexpectedly and for no obvious reason, and without any log messages.

We ought to improve the situation. Consider the following:

Alert operators better when condition is not met, e.g
- Nomad agent report an error or a warning on launch if two nomad agents are interacting with the same agent
- Use consul's lock/sessions to ensure this properly
Remove that restriction and allow multiple nomad agents to connect to the same consul
- When removing querying consul, use labels/tags to filter by nomad agent

[1] https://github.com/hashicorp/nomad/blob/v0.11.0-beta2/website/pages/docs/configuration/consul.mdx#consul-stanza .

shoenig commented 4 years ago

Since this is an error condition that keeps coming up, it does seem like there must be some desire to run in this configuration.

We'll probably need our queries to filter on our own generated IDs though - I suspect inserting tags is going to be unexpected for a lot of environments

tgross commented 4 years ago

Since this is an error condition that keeps coming up, it does seem like there must be some desire to run in this configuration.

I'm pretty sure this configuration breaks Consul service discovery in the general, so it's not just a Nomad-specific misconfiguration.

rf-guo commented 4 years ago

Since this is an error condition that keeps coming up, it does seem like there must be some desire to run in this configuration.

I'm pretty sure this configuration breaks Consul service discovery in the general, so it's not just a Nomad-specific misconfiguration.

By ip address? please review my PR https://github.com/hashicorp/nomad/pull/7597

stale[bot] commented 4 years ago

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

tgross commented 4 years ago

Closed b/c we closed out the discussion in #7819

notnoop commented 4 years ago

I don't think this is ought to close. It's still confusing and better error messages are still appreciated.

In #7918, we decided that we don't want to support multiple nomad agents connecting to the same consul agent using tags - not that we no longer consider the situation confusing.