apiserver received an error that is not an metav1.Status: &errors.errorString{s:"error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not xxx"}

[wang-xiaowu]

Environmental Info: K3s Version: 1.25.16+k3s4

k3s version v1.25.16+k3s4 (ddda2476) go version go1.20.10

Node(s) CPU architecture, OS, and Version: CentOS Linux 7 (Core) 5.4.211-1.el7.elrepo.x86_64

Cluster Configuration: 1 servers, 5 agents

Describe the bug: when i use kubectl command to excute pod operation, i got the error like this : tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.1.4.13, and k3s log shows apiserver received an error that is not an metav1.Status: &errors.errorString{s:"error dialing backend: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.1.4.13"}

Steps To Reproduce:

• Installed K3s: this is the daemon configfile

  ExecStart=/usr/local/bin/k3s \
  server \
      '--egress-selector-mode=disabled' \
      '--node-ip' \
      '10.1.4.13' \
      '--disable' \
      'traefik' \
      '--kubelet-arg' \
      'eviction-hard=imagefs.available<8%,memory.available<500Mi,nodefs.available<8%' \
      '--kubelet-arg' \
      'system-reserved=cpu=1000m,memory=1500Mi' \
      '--kubelet-arg' \
      'max-pods=40' \

• i have tried to add tls-san param, not only /etc/rancher/k3s/config.yaml, but also in the /etc/systemd/system/multi-user.target.wants/k3s.service, update it to 10.1.4.13, then regenerated the secrets/k3s-serving, but all of them didn't work
• and another phenomenon is: only the pods have been deployed on 10.1.4.13 this machine can not operated by kubectl

• Expected behavior:

Actual behavior:

Additional context / logs:

[brandond]

tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.1.4.13"

K3s doesn't generate any certificates that are valid for only the loopback address, but not any other IPs. I also see that you've set the egress-selector mode to disabled; why? Do you perhaps have a http proxy configured in your environment? I'm not sure what exactly the apiserver is talking to here that has this invalid certificate but I don't think it's an internal component.

[wang-xiaowu]

tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.1.4.13"

K3s doesn't generate any certificates that are valid for only the loopback address, but not any other IPs. I also see that you've set the egress-selector mode to disabled; why? Do you perhaps have a http proxy configured in your environment? I'm not sure what exactly the apiserver is talking to here that has this invalid certificate but I don't think it's an internal component.

the reason of disable egress-selector is from another issue : https://github.com/k3s-io/k3s/issues/5897 and there's not http proxy config in my env BTW i have deleted that line in my /etc/systemd/system/multi-user.target.wants/k3s.service and reloaded, but not worked

[wang-xiaowu]

i did a testing, use kubernetes-client to operate k3s with config file, like this, and replace 127.0.0.1 to 10.1.4.13, it worked. i was thinking maybe there's something wrong i ignored inside k3s

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: xxx
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: xxx
    client-key-data: xxx

[wang-xiaowu]

another phenomenon: operation only does not work in the node which metrics-server was deployed after resinstalling the metrics-server, it has worked