Improved message when not authorised to list namespaces

[vgibilmanno]

I can't use kubebox when I only have access to a namespace in a cluster. It would be awesome if I could use kubebox just for the namespace I have access to, ignoring everything outside the namespace.

[astefanutti]

You should be able to use Kubebox just for the namespaces you're granted access to. What error do you face?

[vgibilmanno]

The login fails with "Authentication failed" and I'm being asked for an username and a password. The cluster runs in AWS and I have the AWS credentials set. Because of that I don't really have a username or password to set. Kubectl can be executed perfectly fine.

[astefanutti]

AWS EKS is supposed to be working :(

There have been some issues that may provide some context: #32 #60.

What version of Kubebox do you use? Could you provide the content of your kubeconfig file?

In the meantime, you can run:

$ aws eks get-token --cluster-name [my-cluster] 

To get a token to authenticate with Kubebox.

[vgibilmanno]

Of course :) that's the content of the kubeconfig. I hope it's ok, that I replaced personal information with placeholders. But I didn't change the structure of the content.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://OMITTED.amazonaws.com
  name: SOMENAME
contexts:
- context:
    cluster: SOMENAME
    user: SOMENAME
  name: SOMENAME
current-context: SOMENAME
kind: Config
preferences: {}
users:
- name: SOMENAME
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - SOMEOTHERNAME
      - -r
      - arn:aws:iam::SOMEROLEID:role/SOMEROLE
      command: aws-iam-authenticator
      env: null

I'm using kubebox v0.6.1

How can I replace the token in the login screen? I'm too stupid right now sry

[astefanutti]

Thanks for the details. Nothing obvious so we'll have to try reproducing.

There is a token field in the login window, below the username and password fields. You can paste the token retrieve from AWS CLI.

[johnpoth]

@vgibilmanno perhaps some useful info would be to run the command locally yourself:

aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE and maybe show us the output ? Of course censuring private data. Thanks !

[vgibilmanno]

@johnpoth I get the following

{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1alpha1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2019-11-27T15:51:10Z",
        "token": "SOMETOKEN"
    }
}

[astefanutti]

@johnpoth could it be that the provideAuth method does not get a first token with refresh_token when there is no token information?

[astefanutti]

@johnpoth the refresh_token method returns false when there is no expiration date:

console.log((undefined - Date.now()) < 5000)
> false

So we may have to add a condition to check whether a token exists:

https://github.com/astefanutti/kubebox/blob/f8cb4db7688be5316d7cf87ce3b677911f18755b/lib/auth/exec.js#L14

WDYT?

[johnpoth]

Hi @astefanutti so the expired is always set to now:

ExecAuthProvider ... this.expiry = Date.now();

So the token is always refreshed even if it's still valid on the first run. Therefore I think the problem lies elsewhere but I may be wrong...

@vgibilmanno have you tried pasting the TOKEN you got from running the command into the kubebox login screen ? This should help narrow things down a bit... thanks

[astefanutti]

@johnpoth I've just realised the expiry is set there:

https://github.com/astefanutti/kubebox/blob/f8cb4db7688be5316d7cf87ce3b677911f18755b/lib/config/user.js#L143

It seems it'd be easier to reason about if it would not be initialised... Could it be the login sequence takes less than 5ms so that the check fails?

[vgibilmanno]

@johnpoth I can't paste the token in the login screen. The paste command is not working and I would have to type in the whole token manually. I'm using alpine linux v3.10 in case that's relevant.

[johnpoth]

@astefanutti I hadn't thought about that... judging from the feedback #60 I'm thinking it might be something different ? That being said I don't see what could be causing this from the Kubebox Exec plugin perspective...

It may be interesting for @vgibilmanno to use the Token directly and see if Kubebox can connect to the cluster ?

[astefanutti]

@vgibilmanno could you update your kubeconfig manually and add a token field with the copied token returned by AWS CLI, e.g.:

users:
- name: SOMENAME
  user:
    token: PASTE THERE

Then run Kubebox? The token should be picked-up by Kubebox.

[vgibilmanno]

@astefanutti Doesn't work. The token displayed in the login screen is not the same as I entered in the kubeconfig. My config file is located under /root/.kube/config

[astefanutti]

@vgibilmanno the token may just be truncated in the UI. Have you changed the KUBECONFIG environment variable or does you /root/.kube/config file contain multiple contexts?

[astefanutti]

@vgibilmanno Can you also just try a second time connecting, by just pressing enter when the login widget displays with the error message?

[vgibilmanno]

@astefanutti I tried, but it doesn't work. I'm getting the "authentication failed" message. The token doesn't seem to be the one I define in the kubeconfig. I looked for some substring (last 5 characters) but it doesn't match

[astefanutti]

@vgibilmanno thanks for the feedback. Would you mind checking you have only one context defined in your kubeconfig file?

[vgibilmanno]

@astefanutti There is only one context defined. It's exactly like the one I posted before with the addition, that tere is a token value inside the user field.

[astefanutti]

@vgibilmanno thanks. May I ask, is there a token filled in the login widget when using the original kubeconfig, before adding the token manually to it? Finally could you double check the cluster URL displayed in the login widget is the correct one?

[vgibilmanno]

@astefanutti When I use the original kubeconfig, there is a token filled in the login widget. The cluster URL displayed in the login widget is correct.

[astefanutti]

@vgibilmanno thanks again. Just one last test if you don't mind (sorry). Could you re-try the same test of adding the token manually to your kubeconfig file AND removing the whole exec section? Then check the token is the same as the one displayed in the login widget?

[vgibilmanno]

@astefanutti Now the correct token is displayed, but the authentication fails. I tried pressing enter/log-in multiple times in series

[astefanutti]

OK thanks a lot! So there is a request Kubebox does and that's not authorised for your user. Let me check and come back to you. Thanks again!

[astefanutti]

@vgibilmanno if you could try executing the following commands and provide the results:

$ curl -k -v -H "Authorization: Bearer <TOKEN>" https://OMITTED.amazonaws.com/

$ curl -k -v -H "Authorization: Bearer <TOKEN>" https://OMITTED.amazonaws.com/api

$ curl -k -v -H "Authorization: Bearer <TOKEN>" https://OMITTED.amazonaws.com/api/v1/namespaces

[vgibilmanno]

@astefanutti All 3 requests fail with the following message

*   Trying SOMEIP:443...
* TCP_NODELAY set
* Connected to SOMESERVER.amazonaws.com (SOMEIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: Jan 23 14:39:32 2019 GMT
*  expire date: Nov 17 23:02:45 2020 GMT
*  issuer: CN=kubernetes
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56318da783e0)
> GET / HTTP/2
> Host: SOMEHOST.amazonaws.com
> User-Agent: curl/7.66.0
> Accept: */*
> Authorization: Bearer SOMETOKEN
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 401
< audit-id: 6562ed5f-ef44-4d1b-8210-5c98f0c3cea5
< content-type: application/json
< content-length: 165
< date: Thu, 28 Nov 2019 17:23:36 GMT
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Connection #0 to host SOMEHOST.amazonaws.com left intact

[astefanutti]

@vgibilmanno thanks, just to be sure, could you try with a fresh token that you've just retrieve with aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE?

[vgibilmanno]

@astefanutti I tried but it still fails. I tried the following url too (fails too) curl -k -v -H "Authorization: Bearer <TOKEN>" https://OMITTED.amazonaws.com/api/v1/namespaces/SOMENAMESPACE/pods

[astefanutti]

@vgibilmanno thanks, this is very surprising. Could you try executing the following command and provide the output:

$ kubectl --v=9 get --raw /api/v1/namespaces/SOMENAMESPACE/pods

[vgibilmanno]

@astefanutti This command works. I get the following:

I1129 07:56:44.179968     155 loader.go:359] Config loaded from file /root/.kube/config
I1129 07:56:44.180408     155 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.13.8 (linux/amd64) kubernetes/0c6d31a" 'https://OMITTED.amazonaws.com/api/v1/namespaces/SOMENAMESPACE/pods'
I1129 07:56:45.749509     155 round_trippers.go:438] GET https://OMITTED.amazonaws.com/api/v1/namespaces/SOMENAMESPACE/pods 200 OK in 1569 milliseconds
I1129 07:56:45.749809     155 round_trippers.go:444] Response Headers:
I1129 07:56:45.749821     155 round_trippers.go:447]     Date: Fri, 29 Nov 2019 07:56:45 GMT
I1129 07:56:45.749832     155 round_trippers.go:447]     Audit-Id: f15acf96-5aa7-4f65-a60c-563a94c2bb81
I1129 07:56:45.749840     155 round_trippers.go:447]     Content-Type: application/json
{HUGEJSONCONTENT}

[astefanutti]

@vgibilmanno thanks. Did you remove -H "Authorization: Bearer SOMETOKEN" from the curl command in the output?

Could you just run:

$ curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.13.8 (linux/amd64) kubernetes/0c6d31a" 'https://OMITTED.amazonaws.com/api/v1/namespaces/SOMENAMESPACE/pods'

[vgibilmanno]

@astefanutti

*   Trying SOMEIP:443...
* TCP_NODELAY set
* Connected to OMITTED.amazonaws.com (SOMEIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: Jan 23 14:39:32 2019 GMT
*  expire date: Nov 17 22:59:51 2020 GMT
*  issuer: CN=kubernetes
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x561242df8320)
> GET /api/v1/namespaces/SOMENAMESPACE/pods HTTP/2
> Host: OMITTED.amazonaws.com
> Accept: application/json, */*
> User-Agent: kubectl/v1.13.8 (linux/amd64) kubernetes/0c6d31a
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 403
< audit-id: e42d6da9-82ef-4882-93df-532e4655384f
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 271
< date: Fri, 29 Nov 2019 08:10:38 GMT
<
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"SOMENAMESPACE\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
* Connection #0 to host OMITTED.amazonaws.com left intact

[astefanutti]

@vgibilmanno thanks, it seems logical. I'm surprised there is no -H "Authorization: Bearer SOMETOKEN" output for the curl command when you run kubectl --v=9 get --raw /api/v1/namespaces/SOMENAMESPACE/pods.

I'm running out of idea. Just to be sure, what's the output of:

$ echo $KUBECONFIG

[vgibilmanno]

@astefanutti It's strange, that the curl command works inside of kubectl but the same curl command does not work outside of it.

$ echo $KUBECONFIG

returns nothing

[astefanutti]

@vgibilmanno yes, there is something else that we are missing there.

[johnpoth]

It may be that certificate-authority-data: DATA+OMITTED from @vgibilmanno's Kubernetes config isn't taken into account when executing the curl command, if it's not present in /etc/ssl/certs/ca-certificates.crt - so adding it will micmic more closely what kubetcl does. That being said I don't think it is the root cause as kubebox takes the ca into account

ps: it is normal that the Authorization Http header isn't logged, see https://github.com/kubernetes/client-go/blob/8d0e6f1b7b780ef8c8f81800d504509cd1f68e9f/transport/round_trippers.go#L392

[astefanutti]

@johnpoth I see the Authorization header set when I run the command locally. Not sure about how the CA data could interfere.

[astefanutti]

401 means that the provided token cannot be authenticated to a known identity. It's like the token used is not correct. Maybe there is an encoding issue when Kubebox reads from stdout and parse the response returned from the executed AWS CLI command.

[astefanutti]

@vgibilmanno by any chance, would you be able to clone this repository and run Kubebox from source?

[vgibilmanno]

@astefanutti Yes I'm able to 👍

[astefanutti]

@vgibilmanno awesome! You'll need Node.js installed, so if you can:

• Clone this repository: git clone https://github.com/astefanutti/kubebox.git && cd kubebox
• Then install dependencies: npm i
• Add require('fs').writeFileSync('reponse.json', response); above the following line: https://github.com/astefanutti/kubebox/blob/01b28a2dc33831c635f4577e691c3a2bc977a6a0/lib/auth/exec.js#L52
• Then run node index.js
• Finally cat response.json

Thanks again!

[vgibilmanno]

@astefanutti I get the following in the response.json

{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1alpha1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2019-11-29T11:07:51Z",
        "token": "SOMETOKEN"
    }
}

[astefanutti]

@vgibilmanno thanks. Could you try running:

$ aws-iam-authenticator verify -i <clusterid> -t SOMETOKEN

by taking SOMETOKEN from response.json.

[vgibilmanno]

@astefanutti When I enter the cluster name in I get the following:

could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403)

When I enter the cluster address in I get the following:

could not verify token: input token was not properly formatted: non-whitelisted query parameter "X-Amz-SignedHeade"

[astefanutti]

@vgibilmanno could you try:

$ aws-iam-authenticator verify -i SOMEOTHERNAME -t SOMETOKEN

and provide the output of:

$ env

I suspect there are some environment variables that come into the play.

[vgibilmanno]

@astefanutti So the command again fails with:

could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403)

And have the following entries in env:

HOSTNAME=e39ab07991c2
PYTHON_PIP_VERSION=19.3.1
SHLVL=2
HOME=/root
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/ffe826207a010164265d9cc807978e3604d18ca0/get-pip.py
TERM=xterm
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
PYTHON_VERSION=3.8.0
PWD=/
PYTHON_GET_PIP_SHA256=b86f36cc4345ae87bfd4f10ef6b2dbfa7a872fbff70608a1e43944d283fd0eee

[astefanutti]

Damn, would you mind now trying:

$ aws-iam-authenticator verify -i SOMEOTHERNAME -t `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`

(if you don't have jq, you can copy the token manually)

[vgibilmanno]

@astefanutti &{ARN:arn:aws:sts::SOMEROLEID:assumed-role/SOMEROLE/SOMESESSIONNAME CanonicalARN:arn:aws:iam::SOMEROLEID:role/SOMEROLE AccountID:SOMEACCOUNTID UserID:SOMEUSERID SessionName:SOMESESSIONNAME}

[astefanutti]

@vgibilmanno could you check any difference between the token from response.json and the one returned by aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE?