search

rancher / rke2

https://docs.rke2.io/
Apache License 2.0
1.57k stars 268 forks source link

When using kubectl logs to view the kube-proxy pod logs on the agent node received a "400 Bad Request" error #6842

Closed warnerchen closed 1 month ago

warnerchen commented 1 month ago

Environmental Info: RKE2 Version: v1.26.15+rke2r1 Cluster Configuration: 3 servers + n agents

Describe the bug: In my Rancher, I found that I could not view the kube-proxy pod logs of the agent nodes through the "View Logs" feature (it shows disconnected), but I can view the kube-proxy pod logs of the server node normally.

After that, I tried to view the kube-proxy pod logs on the agent nodes using kubectl -n kube-system logs kube-proxy-xxx from the server node, but it didn't work either.

yyy:/root # kubectl -n kube-system logs kube-proxy-xxx -v 7
I0918 16:48:31.348464    5761 loader.go:373] Config loaded from file:  /etc/rancher/rke2/rke2.yaml
I0918 16:48:31.349079    5761 round_trippers.go:463] GET https://127.0.0.1:6443/api?timeout=32s
I0918 16:48:31.349093    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.349100    5761 round_trippers.go:473]     Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json
I0918 16:48:31.349107    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.357916    5761 round_trippers.go:574] Response Status: 200 OK in 8 milliseconds
I0918 16:48:31.384732    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis?timeout=32s
I0918 16:48:31.384766    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.384790    5761 round_trippers.go:473]     Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json
I0918 16:48:31.384820    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.391537    5761 round_trippers.go:574] Response Status: 200 OK in 6 milliseconds
I0918 16:48:31.393493    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1?timeout=32s
I0918 16:48:31.393516    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393532    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393541    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/apiregistration.k8s.io/v1?timeout=32s
I0918 16:48:31.393562    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/metrics.k8s.io/v1beta1?timeout=32s
I0918 16:48:31.393567    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393591    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393607    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393614    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393626    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393634    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393664    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/apps/v1?timeout=32s
I0918 16:48:31.393674    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393685    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393696    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393705    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/acme.cert-manager.io/v1?timeout=32s
I0918 16:48:31.393734    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393759    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393779    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393797    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/events.k8s.io/v1?timeout=32s
I0918 16:48:31.393805    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393816    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393827    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393837    5761 round_trippers.go:463] GET https://127.0.0.1:6443/api/v1?timeout=32s
I0918 16:48:31.393859    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393875    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393891    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393925    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/storage.k8s.io/v1beta1?timeout=32s
I0918 16:48:31.393933    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393944    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393955    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394075    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/authorization.k8s.io/v1?timeout=32s
I0918 16:48:31.394085    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394096    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394107    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394183    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/autoscaling/v2?timeout=32s
I0918 16:48:31.394191    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394201    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394212    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394318    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/batch/v1?timeout=32s
I0918 16:48:31.394325    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394335    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394346    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394387    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/admissionregistration.k8s.io/v1?timeout=32s
I0918 16:48:31.394414    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394445    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394479    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394520    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/certificates.k8s.io/v1?timeout=32s
I0918 16:48:31.394531    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394540    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394549    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394644    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/cert-manager.io/v1?timeout=32s
I0918 16:48:31.394680    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394695    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394710    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394519    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/catalog.cattle.io/v1?timeout=32s
I0918 16:48:31.394779    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394789    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394800    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394799    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/apiextensions.k8s.io/v1?timeout=32s
I0918 16:48:31.394820    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394839    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394859    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394878    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/compute.functionmesh.io/v1alpha1?timeout=32s
I0918 16:48:31.394886    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394895    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394903    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394976    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/helm.cattle.io/v1?timeout=32s
I0918 16:48:31.394982    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394991    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394999    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395030    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/scheduling.k8s.io/v1?timeout=32s
I0918 16:48:31.395045    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395059    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/k3s.cattle.io/v1?timeout=32s
I0918 16:48:31.395063    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395084    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395223    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/snapshot.storage.k8s.io/v1beta1?timeout=32s
I0918 16:48:31.395230    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395238    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395247    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395311    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/ui.cattle.io/v1?timeout=32s
I0918 16:48:31.395317    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395326    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395334    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395344    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1?timeout=32s
I0918 16:48:31.395352    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395364    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395373    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395391    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/upgrade.cattle.io/v1?timeout=32s
I0918 16:48:31.395398    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395406    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395415    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395442    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/node.k8s.io/v1?timeout=32s
I0918 16:48:31.395449    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395458    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395467    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395476    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/plantcon.bmwgroup.io/v1alpha?timeout=32s
I0918 16:48:31.395483    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395491    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395500    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395531    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/discovery.k8s.io/v1?timeout=32s
I0918 16:48:31.395538    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395547    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395557    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395563    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/admin.pulsar.io/v1alpha1?timeout=32s
I0918 16:48:31.395571    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395579    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395591    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395637    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/policy/v1?timeout=32s
I0918 16:48:31.395652    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395662    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395672    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395689    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/flowcontrol.apiserver.k8s.io/v1beta3?timeout=32s
I0918 16:48:31.395702    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395715    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395730    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395740    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/management.cattle.io/v3?timeout=32s
I0918 16:48:31.395748    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395756    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395765    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.396193    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/flowcontrol.apiserver.k8s.io/v1beta2?timeout=32s
I0918 16:48:31.396218    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396229    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396274    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.396395    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/autoscaling/v1?timeout=32s
I0918 16:48:31.396411    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396440    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396486    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.396607    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/networking.k8s.io/v1?timeout=32s
I0918 16:48:31.396623    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396638    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396653    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.396746    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1?timeout=32s
I0918 16:48:31.396756    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396770    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396784    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395066    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396847    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396869    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395106    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/monitoring.coreos.com/v1?timeout=32s
I0918 16:48:31.396947    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396964    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396985    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395139    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/monitoring.coreos.com/v1alpha1?timeout=32s
I0918 16:48:31.397065    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397087    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397105    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395183    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/snapshot.storage.k8s.io/v1?timeout=32s
I0918 16:48:31.397173    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397191    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397206    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.397367    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/cilium.io/v2alpha1?timeout=32s
I0918 16:48:31.397391    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397421    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397441    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.397558    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/cilium.io/v2?timeout=32s
I0918 16:48:31.397577    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397596    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397615    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.397762    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/cluster.cattle.io/v3?timeout=32s
I0918 16:48:31.397782    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397805    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397830    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393545    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394762    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/storage.k8s.io/v1?timeout=32s
I0918 16:48:31.397950    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397969    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397993    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.398241    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/controller-manager.dvpe-cloud.github.io/v1alpha1?timeout=32s
I0918 16:48:31.398261    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.398282    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.398303    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.398471    5761 round_trippers.go:574] Response Status: 200 OK in 4 milliseconds
I0918 16:48:31.399630    5761 round_trippers.go:574] Response Status: 200 OK in 5 milliseconds
I0918 16:48:31.399677    5761 round_trippers.go:574] Response Status: 200 OK in 5 milliseconds
I0918 16:48:31.400247    5761 round_trippers.go:574] Response Status: 200 OK in 5 milliseconds
I0918 16:48:31.400789    5761 round_trippers.go:574] Response Status: 200 OK in 7 milliseconds
I0918 16:48:31.400881    5761 round_trippers.go:574] Response Status: 200 OK in 7 milliseconds
I0918 16:48:31.401091    5761 round_trippers.go:574] Response Status: 200 OK in 6 milliseconds
I0918 16:48:31.402134    5761 round_trippers.go:574] Response Status: 200 OK in 7 milliseconds
I0918 16:48:31.402168    5761 round_trippers.go:574] Response Status: 200 OK in 6 milliseconds
I0918 16:48:31.402704    5761 round_trippers.go:574] Response Status: 200 OK in 7 milliseconds
I0918 16:48:31.402772    5761 round_trippers.go:574] Response Status: 200 OK in 8 milliseconds
I0918 16:48:31.402802    5761 round_trippers.go:574] Response Status: 200 OK in 9 milliseconds
I0918 16:48:31.402885    5761 round_trippers.go:574] Response Status: 200 OK in 7 milliseconds
I0918 16:48:31.402978    5761 round_trippers.go:574] Response Status: 200 OK in 8 milliseconds
I0918 16:48:31.403098    5761 round_trippers.go:574] Response Status: 200 OK in 8 milliseconds
I0918 16:48:31.403536    5761 round_trippers.go:574] Response Status: 200 OK in 9 milliseconds
I0918 16:48:31.405704    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.405732    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.405756    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.407919    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.407965    5761 round_trippers.go:574] Response Status: 200 OK in 12 milliseconds
I0918 16:48:31.407984    5761 round_trippers.go:574] Response Status: 200 OK in 12 milliseconds
I0918 16:48:31.407987    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408046    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408077    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.408077    5761 round_trippers.go:574] Response Status: 200 OK in 12 milliseconds
I0918 16:48:31.408088    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.408118    5761 round_trippers.go:574] Response Status: 200 OK in 14 milliseconds
I0918 16:48:31.408119    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.408201    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408219    5761 round_trippers.go:574] Response Status: 200 OK in 12 milliseconds
I0918 16:48:31.408233    5761 round_trippers.go:574] Response Status: 200 OK in 12 milliseconds
I0918 16:48:31.408244    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408353    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408396    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408412    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408485    5761 round_trippers.go:574] Response Status: 200 OK in 14 milliseconds
I0918 16:48:31.408525    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.408560    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.408565    5761 round_trippers.go:574] Response Status: 200 OK in 13 milliseconds
I0918 16:48:31.408576    5761 round_trippers.go:574] Response Status: 200 OK in 13 milliseconds
I0918 16:48:31.408781    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.409022    5761 round_trippers.go:574] Response Status: 200 OK in 15 milliseconds
I0918 16:48:31.562466    5761 round_trippers.go:463] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/pods/kube-proxy-xxx
I0918 16:48:31.562508    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.562537    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.562560    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.565911    5761 round_trippers.go:574] Response Status: 200 OK in 3 milliseconds
I0918 16:48:31.571760    5761 round_trippers.go:463] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/pods/kube-proxy-xxx/log?container=kube-proxy
I0918 16:48:31.571787    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.571798    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.571815    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.641024    5761 round_trippers.go:574] Response Status: 400 Bad Request in 69 milliseconds
I0918 16:48:31.645923    5761 helpers.go:246] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "container \"kube-proxy\" in pod \"kube-proxy-xxx\" is not available",
  "reason": "BadRequest",
  "code": 400
}]
Error from server (BadRequest): container "kube-proxy" in pod "kube-proxy-xxx" is not available

However, when I logged into the agent node and used crictl log, I could view the logs normally, so the issue seems to be on the Kubernetes level.

xxx:/root # crictl ps -a | grep kube-proxy
d7b795fbb538b       4a2dd7d23859b       6 weeks ago         Running             kube-proxy                      0                   7f83862ecf3a1       kube-proxy-xxx

xxx:/root # ps aux | grep kube-proxy 
root     16955  0.0  0.1 1280240 55640 ?       Ssl  Aug05  30:34 kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=xxx --kubeconfig=/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig --proxy-mode=iptables
root     29007  0.0  0.0   5644   816 pts/0    S+   15:37   0:00 grep kube-proxy

xxx:/root # crictl logs $kube_proxy_container_id
I0805 05:38:27.037153       1 server.go:224] "Warning, all flags other than --config, --write-config-to, and --cleanup are deprecated, please begin using a config file ASAP"
I0805 05:38:27.054925       1 node.go:163] Successfully retrieved node IP: 10.117.158.31
I0805 05:38:27.054998       1 server_others.go:109] "Detected node IP" address="10.117.158.31"
I0805 05:38:27.105372       1 server_others.go:178] "Using iptables Proxier"
I0805 05:38:27.105491       1 server_others.go:185] "kube-proxy running in dual-stack mode" ipFamily=IPv4
I0805 05:38:27.105516       1 server_others.go:186] "Creating dualStackProxier for iptables"
I0805 05:38:27.105552       1 server_others.go:468] "Detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6"
I0805 05:38:27.105612       1 proxier.go:242] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses"
I0805 05:38:27.106246       1 server.go:656] "Version info" version="v1.26.15+rke2r1"
I0805 05:38:27.106280       1 server.go:658] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0805 05:38:27.107138       1 config.go:226] "Starting endpoint slice config controller"
I0805 05:38:27.107220       1 shared_informer.go:270] Waiting for caches to sync for endpoint slice config
I0805 05:38:27.107223       1 config.go:317] "Starting service config controller"
I0805 05:38:27.107256       1 shared_informer.go:270] Waiting for caches to sync for service config
I0805 05:38:27.107223       1 config.go:444] "Starting node config controller"
I0805 05:38:27.107319       1 shared_informer.go:270] Waiting for caches to sync for node config
I0805 05:38:27.207742       1 shared_informer.go:277] Caches are synced for endpoint slice config
I0805 05:38:27.207902       1 shared_informer.go:277] Caches are synced for node config
I0805 05:38:27.208044       1 shared_informer.go:277] Caches are synced for service config

xxx:/root # cat /var/log/pods/kube-system_kube-proxy-xxx_c16282c8793e789dcf001594d2941864/kube-proxy/0.log 
2024-08-05T13:38:27.037609427+08:00 stderr F I0805 05:38:27.037153       1 server.go:224] "Warning, all flags other than --config, --write-config-to, and --cleanup are deprecated, please begin using a config file ASAP"
2024-08-05T13:38:27.055400955+08:00 stderr F I0805 05:38:27.054925       1 node.go:163] Successfully retrieved node IP: 10.117.158.31
2024-08-05T13:38:27.055451729+08:00 stderr F I0805 05:38:27.054998       1 server_others.go:109] "Detected node IP" address="10.117.158.31"
2024-08-05T13:38:27.105839301+08:00 stderr F I0805 05:38:27.105372       1 server_others.go:178] "Using iptables Proxier"
2024-08-05T13:38:27.105877094+08:00 stderr F I0805 05:38:27.105491       1 server_others.go:185] "kube-proxy running in dual-stack mode" ipFamily=IPv4
2024-08-05T13:38:27.105882616+08:00 stderr F I0805 05:38:27.105516       1 server_others.go:186] "Creating dualStackProxier for iptables"
2024-08-05T13:38:27.105896168+08:00 stderr F I0805 05:38:27.105552       1 server_others.go:468] "Detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6"
2024-08-05T13:38:27.105902017+08:00 stderr F I0805 05:38:27.105612       1 proxier.go:242] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses"
2024-08-05T13:38:27.106399454+08:00 stderr F I0805 05:38:27.106246       1 server.go:656] "Version info" version="v1.26.15+rke2r1"
2024-08-05T13:38:27.106422753+08:00 stderr F I0805 05:38:27.106280       1 server.go:658] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
2024-08-05T13:38:27.107445169+08:00 stderr F I0805 05:38:27.107138       1 config.go:226] "Starting endpoint slice config controller"
2024-08-05T13:38:27.107453733+08:00 stderr F I0805 05:38:27.107220       1 shared_informer.go:270] Waiting for caches to sync for endpoint slice config
2024-08-05T13:38:27.107458336+08:00 stderr F I0805 05:38:27.107223       1 config.go:317] "Starting service config controller"
2024-08-05T13:38:27.107462883+08:00 stderr F I0805 05:38:27.107256       1 shared_informer.go:270] Waiting for caches to sync for service config
2024-08-05T13:38:27.107468516+08:00 stderr F I0805 05:38:27.107223       1 config.go:444] "Starting node config controller"
2024-08-05T13:38:27.107473363+08:00 stderr F I0805 05:38:27.107319       1 shared_informer.go:270] Waiting for caches to sync for node config
2024-08-05T13:38:27.208175431+08:00 stderr F I0805 05:38:27.207742       1 shared_informer.go:277] Caches are synced for endpoint slice config
2024-08-05T13:38:27.208214556+08:00 stderr F I0805 05:38:27.207902       1 shared_informer.go:277] Caches are synced for node config
2024-08-05T13:38:27.208231535+08:00 stderr F I0805 05:38:27.208044       1 shared_informer.go:277] Caches are synced for service config

xxx:/root # curl 127.0.0.1:10256/healthz 
{"lastUpdated": "2024-09-19 03:00:31.379390098 +0000 UTC m=+3878524.370184849","currentTime": "2024-09-19 03:00:31.379390098 +0000 UTC m=+3878524.370184849"}

After I restarted the rke2-agent and recreated the kube-proxy pod, everything returned to normal.