Open nwmqpa opened 8 months ago
Small update as I've come to understand a bit more of the issue after looking at the source code:
The SI token should be sent to envoy through the envoy_bootstrap_hook
, and should be notified here
However, it doesn't seem like any SI token has been delivered (despite one having been created on the consul cluster). So it would appear that no tokens could have been received by this function
As the documentation led me to believe the token should be available as soon as the sids_hooks
has been completed, I've taken a look at the code initializing the hook and I've come to understand that the issue at hand is that, without specifying a token on the consul config part of the nomad agent configuration, nomad would believe consul ACL are not enabled, and therefore skipping the sids_hook
This is an issue in setups like mines where default consul token are set at the consul agent level, and not in the nomad config.
I'm sure there is a more surefire way to identify wether consul ACL are enabled, because as far as I can see, the specified consul token is used nowhere in the SI token derivation, and could be avoided entirely
Hi @nwmqpa! Just for my clarification, did you have the same problem prior to Nomad 1.7 and/or without Workload Identity?
I've taken a look at the code initializing the hook and I've come to understand that the issue at hand is that, without specifying a token on the consul config part of the nomad agent configuration, nomad would believe consul ACL are not enabled, and therefore skipping the sids_hook
This is an issue in setups like mines where default consul token are set at the consul agent level, and not in the nomad config.
That all makes sense to me but if I look at the same code in 1.6.5 (ref task_runner_hooks.go#L142
) it looks like we had the same behavior. So that's why I'm asking about whether you saw this pre-1.7. But either way it seems like a bug; we should be checking whether the Consul successfully fingerprinted and not looking at the token. Or better yet we should run sids_hook
unconditionally for the Connect task and let it fail gracefully in cases where Consul ACLs aren't configured.
Hello @tgross , I haven't noticed this before migrating to workload identity, but we are in the process of migrating our clusters to the newer workload identities for both consul and vault, and are therefore doing quite a lot of new things at once.
Beforehand, our default was to store consul token in the nomad agent configuration, as we weren't using mTLS, and thought it would be more secure.
So I assume this behaviour has always been here, and is just now coming to light.
I do believe your idea about using the sids_hook
and let it gracefully fail would be better suited, given the default is to already gracefully fail when attempting to get the si_token
Thanks for the context @nwmqpa. I've been a little swamped the last couple weeks, but I'll try to see if I can build out a minimal repro and get some motion into fixing this.
We have the same problem here. Is there any update regarding a fix?
Maybe related, maybe a separate issue, but whilst with our setup we can initially bootstrap and get services up and running. Once the service identity token renews it appears that Envoy is unaware of this and continues trying to use an expired token.
Manually restarting the sidecar task fixes this.
[./source/extensions/config_subscription/grpc/grpc_stream.h:152] DeltaAggregatedResources gRPC config stream to local_agent closed: 16, unauthenticated: ACL not found
Edit: turns out this was because we were renewing Nomad's Consul token periodically, and this is what Envoy uses to bootstrap? See NOMAD-579. Switching to a static Consul token for Nomad resolved this.
I haven't had time to get back to this, so surfacing it for roadmapping so we can get some eyes on it.
Nomad version
Operating system and Environment details
Issue
Envoy proxy doesn't seem to use the service identity token to connect with the gRPC interface, which cause the proxy to never bootstrap
Service identity is properly set up and a token has been issued by consul. No task identity is used
Reproduction steps
Expected Result
Envoy should be able to bootstrap correctly and the consul healthchecks should pass
Actual Result
Envoy isn't correctly bootstrapped and the service is not marked as healthy
Job file (if appropriate)
Nomad Server Config
Nomad Client config
Consul agent config
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
Job Log
Connect Proxy log