GKE authentication support

[srossross]

This example fails for me when using a GKE cluster, but works fine for minikube:

const cfg = Api.config.fromKubeconfig();
const core = new Api.Core(cfg);
core.ns.rc('http-rc').get(print);

...
{ Error: Unauthorized
    at /Users/sean/workspace/go/src/github.com/srossross/GitBotHub/githubapp/node_modules/kubernetes-client/lib/base.js:11:21
    at Request.request [as _callback] (/Users/sean/workspace/go/src/github.com/srossross/GitBotHub/githubapp/node_modules/kubernetes-client/lib/request.js:65:7)
    at Request.self.callback (/Users/sean/workspace/go/src/github.com/srossross/GitBotHub/githubapp/node_modules/request/request.js:186:22)
    at Request.emit (events.js:160:13)
    at Request.<anonymous> (/Users/sean/workspace/go/src/github.com/srossross/GitBotHub/githubapp/node_modules/request/request.js:1163:10)
    at Request.emit (events.js:160:13)
    at IncomingMessage.<anonymous> (/Users/sean/workspace/go/src/github.com/srossross/GitBotHub/githubapp/node_modules/request/request.js:1085:12)
    at Object.onceWrapper (events.js:255:19)
    at IncomingMessage.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12) code: 401 }

Are there instructions on how to authenticate with a GKE cluster?

[ekryski]

Also having the same issue. For some reason it doesn't authenticate with GKE when using the default ~/.kube/config. When calling out to the k8s API it thinks it's an anonymous user making API calls.

Any insight would be great. 😄

[silasbw]

Yikes! Sounds like a bug. I'll try to look into it, but it's a busy week. Can you share the result of fromKubeConfig() (obfuscating secrets, etc)?

[ekryski]

It looks correct, that's what it throwing me off...

{ url: 'https://<our cluster IP>',
  auth:
   { bearer: '<our bearer token>' },
  ca: '-----BEGIN CERTIFICATE-----\n<OUR CA CERT>-----END CERTIFICATE-----\n',
  insecureSkipTlsVerify: false,
  key: undefined,
  cert: undefined,
  promises: true }

FWIW we're running k8s v1.9.2 on GKE.

[ekryski]

I tried curling with the same bearer token from my kube config and I also get an unauthorized error. However, connecting to localhost:8001 when running kubectl proxy totally works. So strange, might not be an issue with this after all... rather something weird going on with our auth credentials.

I'm not sure how kubectl proxy pulls credentials but I'm wondering if our IAM roles are correct but because we haven't created RBAC roles for our users inside k8s that maybe it's not working??

[ekryski]

I lied, it does work when curling. Turns out you need to set insecureSkipTlsVerify: true, so I believe this is a non-issue.

Is there a way to have the cert included in request so that we can avoid MITM attacks? Possibly here? https://github.com/request/request-promise/issues/225#issuecomment-332292500

Although, that seems like it might be doing the same thing really...

[ekryski]

Actually I was mistaken...

Just adding insecureSkipTlsVerify: true doesn't work. It's an RBAC related issue. When I created a new service account with cluster-admin permissions and then grabbed the access token from that service account and used it to curl it worked just fine.

I then added it manually to the config options and it also worked. So there is something weird going on where the IAM user I'm authenticating with works with RBAC via kubectl but not when curling or using kubernetes-client.

[ekryski]

Ah ha! If you are authenticating using your gcloud credentials (ie. the default setup) then you need to create a service account for your email address so that k8s RBAC knows what your Google Cloud credentials have access to. It was buried in these Google docs.

You can run this:

kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user $(gcloud config get-value account)

[silasbw]

So, were you getting a similar error when trying to use kubectl @ekryski?

I'm trying to understand if this issue is specific to kubernetes-client. Thanks for helping track this down.

[roychri]

Hello, I am not sure if this is related, but I am getting Unauthorized when its been a while that I have used a kubectl command. I suspect that's because the token in ~/.kube/config is old and expired. Whenever I get this error, I simply run kubectl get pods and I suspect that has the effect of renewing the token and then my script using kubernetes-client starts working again. Is this related?

I am not using RBAC but I did authenticating using gcloud container clusters get-credentials mycluster --zone $ZONE --project $PROJECTID

[silasbw]

Thanks @roychri. Can you confirm what effect kubectl get pods has on you kubeconfig? E.g., updates access-token and expiry.

[jaredallard]

Experiencing this too, can confirm @roychri's analysis, it's definitely expired credentials. access-token and expiry are updated when running kubectl get pods.

[djmc]

I also use the gcloud container clusters get-credentials ... way of accessing our Kubernetes cluster, and it appears to have set up my kubeconfig differently from @roychri's.

users:
- name: ...
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: /bin/gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

If I run {cmd-path} {cmd-args} I get a JSON output that contains the expiry and access token at the paths specified. I can't work out how to change it to a general Kubernetes set up either, nor can I find out how to access the contained info some other programmatic way without having to spawn that container.

When I looked at the source, it appears to be looking for the auth-provider, but expects the access token to already be present.

[jaredallard]

I've started work on this in https://github.com/jaredallard/kubernetes-client... will post a PR # when it's more established.

It'll implement both GKE & IDP authentication by adding support for cmd-path and IDP related configuration.

[silasbw]

Published @jaredallard's fix in kubernetes-client@5.2.0. Thanks @jaredallard and the folks who reported this!

[djmc]

Hi, I had a try of the new 5.2.0 client and something is still a bit amiss.

Here's the function I'm trying to call,

export default async function getClient() {
  const Client = kubernetes.Client;
  const inst = new Client({ config: kubernetes.config.fromKubeconfig() });
  await inst.loadSpec();
  return inst;
}

Yet I get the following error,

Error: no auth mechanism defined
    at Auth.onRequest (.../node_modules/request/lib/auth.js:132:32)
    at Request.auth (.../node_modules/request/request.js:1349:14)
    at Request.init (.../node_modules/request/request.js:375:10)
    at new Request .../node_modules/request/request.js:128:8)
    at request (.../node_modules/request/index.js:53:10)
    at Request.request (.../node_modules/kubernetes-client/lib/request.js:103:12)
    at Promise (.../node_modules/kubernetes-client/lib/swagger-client.js:108:17)
    at new Promise (<anonymous>)
    at Component.loadSpec (.../node_modules/kubernetes-client/lib/swagger-client.js:107:12)
    at .../index.js:57:47

I had a look and the auth refresh code only seems to happen when you use callbacks, but when tweaking the code myself to work for promises, I couldn't get it to work either. Am I missing something?

[jaredallard]

@djmc What's the contents of your ~/.kube/config? (strip sensitive info!)

[djmc]

@jaredallard sure, the authentication is the same as my comment prior. But here's the full config without sensitive info,

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: REDACTED
  name: gke_..._europe-west1-c_cluster2
contexts:
- context:
    cluster: gke_..._europe-west1-c_cluster2
    user: gke_..._europe-west1-c_cluster2
  name: gke_..._europe-west1-c_cluster2
current-context: gke_..._europe-west1-c_cluster2
kind: Config
preferences: {}
users:
- name: gke_..._europe-west1-c_cluster2
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: /bin/google-cloud-sdk/bin/gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'

[ekryski]

Gah! Sorry @silasbw. Been bad at checking my Github notifications...

So, were you getting a similar error when trying to use kubectl @ekryski?

I don't totally remember but it definitely wasn't an issue with this module per se.

Authenticating to Kube is different than RBAC. Your Google IAM creds don't grant you RBAC permissions. You need to create service accounts with the appropriate RBAC permissions. I think the issue was still the same with kubectl but it might also be that kubectl interactions are run as the cluster-admin role by default. I can't remember.

[dorshay6]

Hi, I had a try of the new 5.2.0 client and something is still a bit amiss.

Here's the function I'm trying to call,

export default async function getClient() {
  const Client = kubernetes.Client;
  const inst = new Client({ config: kubernetes.config.fromKubeconfig() });
  await inst.loadSpec();
  return inst;
}

Yet I get the following error,

Error: no auth mechanism defined
    at Auth.onRequest (.../node_modules/request/lib/auth.js:132:32)
    at Request.auth (.../node_modules/request/request.js:1349:14)
    at Request.init (.../node_modules/request/request.js:375:10)
    at new Request .../node_modules/request/request.js:128:8)
    at request (.../node_modules/request/index.js:53:10)
    at Request.request (.../node_modules/kubernetes-client/lib/request.js:103:12)
    at Promise (.../node_modules/kubernetes-client/lib/swagger-client.js:108:17)
    at new Promise (<anonymous>)
    at Component.loadSpec (.../node_modules/kubernetes-client/lib/swagger-client.js:107:12)
    at .../index.js:57:47

I had a look and the auth refresh code only seems to happen when you use callbacks, but when tweaking the code myself to work for promises, I couldn't get it to work either. Am I missing something?

Same issue here