search

hashicorp / vault-plugin-auth-kubernetes

Vault authentication plugin for Kubernetes Service Accounts
https://www.vaultproject.io/docs/auth/kubernetes.html
Mozilla Public License 2.0
206 stars 62 forks source link

Unable to make Kubernetes requests: no such host #194

Closed roy-work closed 1 year ago

roy-work commented 1 year ago

We are seeing the following in our logs:

vault-1 vault 2023-05-12T20:56:43.996Z [DEBUG] auth.kubernetes.auth_kubernetes_689297a0: login unauthorized: err="Post \"https://kubernetes:443/apis/authentication.k8s.io/v1/tokenreviews\": dial tcp: lookup kubernetes on 10.0.128.10:53: no such host"

The error is correct; there is no such host. How/why is Vault attempting to connect to kubernetes:443? (This is the wrong host: the k8s API is kubernetes.default.svc, see the documentation.)

(Weirdly we have two Vaults, in two NSes, one which is working with this plugin, and one which isn't, and has this error. I am confused as to why they're behaving differently, as their configs are fairly similar.)

roy-work commented 1 year ago

Ugh, alright, so the root cause here is PEBKAC. Mea culpa.

But there are two thing in this plugin that make this really painful to figure that out:

  1. 168 — this makes it impossible to see that auth is failing for problems unrelated to the presented credentials.

  2. The configuration pane for the auth plugin doesn't show all of the settings. We were comparing a known good Vault cluster and the problem cluster's configuration settings for this auth plugin, and since they were identical, we discounted misconfiguration as a possibility. It's not until you attempt to edit the configuration settings that it becomes apparent that there are identical settings, and that in our case, that they're not set the same.