search

kubernetes / kubectl

Issue tracker and mirror of kubectl code
Apache License 2.0
2.75k stars 896 forks source link

rsync via kubectl hangs #1617

Open chrkspr opened 1 week ago

chrkspr commented 1 week ago

What happened:

rsync with kubectl as remote shell gets stuck. After transfering a couple of files nothing happens. The sync was canceled after 20 minutes. When repeating the command it might get stuck after a different file. The files are mostly smaller than 500kb, some have 1 - 100MB. The command was run on one of the kubernetes master nodes. The vm of the master node and the vm where the pod is running are on the same vmware esx host. So it shouldn't be a problem caused by a network component. Encreasing the debug level in the kube-apiserver and the kubelet to 9 didn't reveal any problems. When the rsync command is cancel via ctrl+c, the rsync server command in the pod is not terminated. Sometimes the rsync finishes without any problems.

rsync command:

rsync -aOv --progress --blocking-io --rsh="kubectl exec my-pod-name -c my-container-name -n my-namespace -i -- " rsync:/mnt/content/ /mnt

What you expected to happen:

About 45.000 (8.4GB) should be transfered without problems.

How to reproduce it (as minimally and precisely as possible):

I didn't try, but it should be reproducable by creating lot's of random files (ca. 45.000) with a size of 500kb.

Anything else we need to know?:

The virus scanner was disabled.

Environment:

kubernetes version:

$ kubectl version
Client Version: v1.30.1+rke2r1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1+rke2r1

os: Red Hat Enterprise Linux 8.10 (Ootpa) on VMware vSphere 8

k8s-ci-robot commented 1 week ago

This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
ardaguclu commented 6 days ago

SPDY pings cause such problems, if the connection remains open longer (for reference https://github.com/kubernetes/kubectl/issues/1606#issuecomment-2149219184). I'd recommend testing the same command execution after websockets are in beta in next releases.