[BUG] k3s proxied downstream cluster does not work on v1.24.4+k3s1 but does work on v1.24.6+k3s1

[slickwarren]

Rancher Server Setup

• Rancher version: v2.6-head (e139976)
• Installation option (Docker install/Helm Chart): docker
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): n/a
• Proxy/Cert Details:n/a

Information about the Cluster

• Kubernetes version: v1.24.4+k3s1
• Cluster Type (Local/Downstream): downstream
  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): custom , proxied cluster

    k3s proxied downstream cluster does not work on v1.24.4+k3s1 but does work on v1.24.6+k3s1

    To Reproduce

    • deploy proxy for use in the downstream clsuter
    • in rancher, deploy a custom cluster and specify HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables and use the latest k3s version available
    • connect a node to the custom cluster that is able to reach the proxy you created in an earlier step.

    Result node makes an initial connection, but never gets to an active state.

    Expected Result

    node should be able to register with rancher from behind the proxy

    Screenshots

    Additional context

    • found when testing: https://github.com/rancher/rancher/issues/39068
    • does not happen on 1.24.6+k3sr1

    rancher-system-agent logs:

    level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"
    rancher-system-agent[2790]: time="2022-10-12T18:09:47Z" level=info msg="[82a072010f1716ef0f3c866533fcbef07010c7ce199cc7c51a6d16b7d20dce0a_0:stdout]: Name Location Size Created"
    rancher-system-agent[2790]: time="2022-10-12T18:09:47Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"

    k3s logs just keep spamming this message:

    k3s[2936]: I1012 18:50:27.610159    2936 scope.go:110] "RemoveContainer" containerID="c9617aee67a6aae3a26ef124208a83bfa2dbc9029b0dbb0b570389a4eb9000e0"
    k3s[2936]: I1012 18:50:36.190489    2936 scope.go:110] "RemoveContainer" containerID="c9617aee67a6aae3a26ef124208a83bfa2dbc9029b0dbb0b570389a4eb9000e0"
    k3s[2936]: I1012 18:50:36.191032    2936 scope.go:110] "RemoveContainer" containerID="704d2ab8b106ef5fb27b04e75341bfcc47a258dbb15649986c51a00f4db90a22"
    k3s[2936]: E1012 18:50:36.191476    2936 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cluster-register\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=cluster-register pod=cattle-cluster-agent-6f55b75c86-ppcq6_cattle-system(94520d8a-72e0-427f-b45b-b6842edeeebc)\"" pod="cattle-system/cattle-cluster-agent-6f55b75c86-ppcq6" podUID=94520d8a-72e0-427f-b45b-b6842edeeebc
    k3s[2936]: I1012 18:50:50.609819    2936 scope.go:110] "RemoveContainer" containerID="704d2ab8b106ef5fb27b04e75341bfcc47a258dbb15649986c51a00f4db90a22"

[Sahota1225]

proxy is broken and a fix will be available with a later version

[Sahota1225]

@slickwarren can you please test this with 1.24.9 and see if the issue still exist?

[snasovich]

@slickwarren , moving it "To Test" to check the above. If it's not reproducible on 1.24.9 (the latest to be in 2.7.x), let's close it. We're not going to fix 1.24.4 retroactively.

[slickwarren]

confirmed, this is working on v1.24.8+k3s1. will close this issue