velero CLI fails to authenticate with complext KUBECONFIG

[rombert]

What steps did you take and what happened:

I am using the velero CLI with a KUBECONFIG env variable set to multiple files, e.g.

$HOME/.config/kube/cluster001-kubeconfig.yaml:$HOME/.config/kube/foo.yaml:$HOME/.config/kube/homelab.yaml:

The CLI is unable to authenticate by default:

$ velero version
Client:
    Version: v1.2.0
    Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c
<error getting server version: Unauthorized>

On the other hand, when settubg the env variable to a single file - which is the cluster where velero is deployed - velero works

$ KUBECONFIG=~/.config/kube/cluster001-kubeconfig.yaml velero version
Client:
    Version: v1.2.0
    Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c
Server:
    Version: v1.1.0

What did you expect to happen:

The CLI tool should support the same type of KUBECONFIG setings that kubectl does and should

The output of the following commands will help us better understand what's going on:

(not applicable)

Environment:

• Velero version (use velero version):

  Client:
  Version: v1.2.0
  Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c
  Server:
  Version: v1.1.0

• Velero features (use velero client config get features):

  features: <NOT SET>

• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-16T12:00:00Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:

not sure

• Cloud provider or hardware configuration:

Linode Kubernetes Engine

• OS (e.g. from /etc/os-release):

  NAME="openSUSE Tumbleweed"
  # VERSION="20191128"
  ID="opensuse-tumbleweed"
  ID_LIKE="opensuse suse"
  VERSION_ID="20191128"
  PRETTY_NAME="openSUSE Tumbleweed"
  ANSI_COLOR="0;32"
  CPE_NAME="cpe:/o:opensuse:tumbleweed:20191128"
  BUG_REPORT_URL="https://bugs.opensuse.org"
  HOME_URL="https://www.opensuse.org/"
  LOGO="distributor-logo"

[markrity]

@rombert I have just tested this with similar configuration to yours, for it me it works fine , do you still have this issue?

[rombert]

@markrity - I'll double-check and let you know

[rombert]

@markrity - this still fails for me. In the meantime I upgraded kubectl to 1.17.0, but that did not help.

I tried debugging a bit myself, and have some extra information, maybe that rings some bells:

• I have 3 kubeconfig files, let's call them A,B,C
• the kubeconfig file which references the cluster with the velero instance is A

The following scenarios work:

• KUBECONFIG=A
• KUBECONFIG=A,B

The following scenarios do not work:

• KUBECONFIG=A,B,C
• KUBECONFIG=A,C

I ran the velero -v 9 version command for both working and failing scenarios, and the HTTP calls seem to be the same for both. Notably, the server IP is the correct one.

The C file references a development cluster running on a private network on my machine, e.g.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <REDACTED>
    server: https://10.24.0.70:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: datadog
    user: kubernetes-admin
  name: homelab
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: <REDACTED>
    client-key-data: <REDACTED>

A couple of questions:

• is there a way to also log authentication data with velero? I did not see with -v 9.
• do you see anything in the kubeconfig file above that could trigger such an authentication problem?
• could the fact that the cluster defined in C is stopped be causing problems?

Thanks!

[markrity]

@rombert The only scenario that is missing in my point of view is KUBECONFIG=C. Because it looks like that it is the common value for a failure.

I will take a deeper look soon, thanks for providing more information !

[rombert]

@markrity - I did not try that since the cluster is configured with file A, so using C would not lead to a meaningful result.

[nrb]

@rombert @markrity Thank you both for looking into this!

Velero's been updated with new Kubernetes APIs since January - is this still happening with v1.4.x and v1.5.x? Admittedly I'm not running against multiple clusters, but if it's still happening, I'll get it allocated into the backlog.

[rombert]

@nrb - I just checked and the problem does not seem to be there anymore

$ velero version
Client:
    Version: v1.4.2
    Git commit: 56a08a4d695d893f0863f697c2f926e27d70c0c5
Server:
    Version: v1.3.1

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-15T00:00:00Z", GoVersion:"go1.15.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.12", GitCommit:"5ec472285121eb6c451e515bc0a7201413872fa3", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:12Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

[nrb]

@rombert Interesting, thanks! I'm going to close this issue out for now. Of course, feel free to comment again if you run into further problems.

[STYNET]

I use Velero v1.8.1 and i am a same problem.

When I use a multiples contexts :

$ echo $KUBECONFIG 
~/.kube/config:~/.kube/config:/home/xxxxx/Documents/XXX/k8s-ovh/kubeconfig.yml

I have error connect :

$ velero debug
.2022/06/17 14:38:19 Collecting velero resources in namespace: velero
An error occurred: exec failed: Traceback (most recent call last):
  velero-debug-collector:23:13: in <toplevel>
  <builtin>: in kube_get
Error: could not initialize search client: invalid configuration: [context was not found for specified context: kubernetes-admin@XXX, cluster has no server defined]

But, the context error message disappears if i reconfigure my KUBECONFIG with only one context : $ export KUBECONFIG='/home/xxxxx/Documents/XXX/k8s-ovh/kubeconfig.yml'

$ velero debug 
2022/06/17 14:54:32 Collecting velero resources in namespace: velero
2022/06/17 14:54:34 Collecting velero deployment logs in namespace: velero
2022/06/17 14:54:35 Generated debug information bundle: ~/bundle-2022-06-17-14-54-32.tar.gz

It's a bug or bad use ?