command: '[/tmp/static-tcpdump -i any -U -w - ]' executing successfully exitCode: '126'

[jmvbxx]

After running kubectl sniff <POD> -n <NAMESPACE> I get the following output with error:

INFO[0000] sniffing method: upload static tcpdump
INFO[0000] using tcpdump path at: '/Users/jason/.krew/store/sniff/v1.4.2/static-tcpdump'
INFO[0001] no container specified, taking first container we found in pod.
INFO[0001] selected container: 'container1'
INFO[0001] sniffing on pod: '<POD>' [namespace: '<NAMESPACE>', container: 'container1', filter: '', interface: 'any']
INFO[0001] uploading static tcpdump binary from: '/Users/jason/.krew/store/sniff/v1.4.2/static-tcpdump' to: '/tmp/static-tcpdump'
INFO[0001] uploading file: '/Users/jason/.krew/store/sniff/v1.4.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'container1'
INFO[0001] executing command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' on container: 'container1', pod: '<POD>', namespace: '<NAMESPACE>'
INFO[0002] command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' executing successfully exitCode: '2', stdErr :'ls: cannot access /tmp/static-tcpdump: No such file or directory
'
INFO[0002] file not found on: '/tmp/static-tcpdump', starting to upload
INFO[0003] tcpdump uploaded successfully
INFO[0003] spawning wireshark!
E0717 12:37:48.123789   28353 v2.go:105] write tcp 192.168.1.9:53321->23.20.184.138:443: write: broken pipe
INFO[0003] start sniffing on remote container
INFO[0003] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'container1', pod: '<POD>', namespace: '<NAMESPACE>'
INFO[0004] command: '[/tmp/static-tcpdump -i any -U -w - ]' executing successfully exitCode: '126', stdErr :''
ERRO[0004] failed to start remote sniffing, stopping wireshark  error="executing sniffer failed, exit code: '126'"
INFO[0004] starting sniffer cleanup
INFO[0004] sniffer cleanup completed successfully
Error: signal: killed

Any thoughts on what might be causing this?

[kppullin]

Have you tried using the -v (verbose) flag or the -p (privileged, start another container) flag?

I ran across a similar failure, and with the verbose logs I saw that the upload of tcpdump failed due to the container having a read-only filesystem. Spawning a separate debug container via the -p flag let me work around this.

[slient2010]

Maybe you should use -c (specified container) to specify a container which you want to observe.

[bcm1992]

I had the same exit code 126.

INFO[0000] command: '[/tmp/static-tcpdump -i any -U -w - ]' executing successfully exitCode: '126', stdErr :'' 
ERRO[0000] failed to start remote sniffing, stopping wireshark  error="executing sniffer failed, exit code: '126'"

In my case, the target cluster is ARM64 (Rasberry Pi), and compiling static-tcpdump on an ARM host fixed the issue.

1. Copile static-tcpdump on an ARM64 linux. ( git clone and make static-tcpdump )
2. Copy the static file to the target pod. ( kubectl cp ./static-tcpdump <target-pod>:/tmp/static-tcpdump )
3. Run kubectl sniff to the pod

[bostrt]

Hi all, Are any of you still hitting this issue? I've not been able to reproduce anything similar.

[moonape1226]

I was hitting the same error but using -p flag does help. Thanks @kppullin

[pcgeek86]

I have the same problem. Windows 11.

time="2022-02-16T18:08:46-07:00" level=info msg="using tcpdump path at: 'C:\\Users\\TrevorSullivan\\scoop\\persist\\krew\\.krew\\store\\sniff\\v1.6.2\\static-tcpdump'"
time="2022-02-16T18:08:48-07:00" level=info msg="no container specified, taking first container we found in pod."
time="2022-02-16T18:08:48-07:00" level=info msg="selected container: 'web01'"
time="2022-02-16T18:08:48-07:00" level=info msg="sniffing method: upload static tcpdump"
time="2022-02-16T18:08:48-07:00" level=info msg="sniffing on pod: 'web01' [namespace: 'nginx', container: 'web01', filter: '', interface: 'any']"
time="2022-02-16T18:08:48-07:00" level=info msg="uploading static tcpdump binary from: 'C:\\Users\\TrevorSullivan\\scoop\\persist\\krew\\.krew\\store\\sniff\\v1.6.2\\static-tcpdump' to: '/tmp/static-tcpdump'"
time="2022-02-16T18:08:48-07:00" level=info msg="uploading file: 'C:\\Users\\TrevorSullivan\\scoop\\persist\\krew\\.krew\\store\\sniff\\v1.6.2\\static-tcpdump' to '/tmp/static-tcpdump' on container: 'web01'"
time="2022-02-16T18:08:48-07:00" level=info msg="executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'web01', pod: 'web01', namespace: 'nginx'"
time="2022-02-16T18:08:49-07:00" level=info msg="command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :''"
time="2022-02-16T18:08:49-07:00" level=info msg="file found: ''"
time="2022-02-16T18:08:49-07:00" level=info msg="file was already found on remote pod"
time="2022-02-16T18:08:49-07:00" level=info msg="tcpdump uploaded successfully"
time="2022-02-16T18:08:49-07:00" level=info msg="spawning wireshark!"
time="2022-02-16T18:08:49-07:00" level=info msg="start sniffing on remote container"
time="2022-02-16T18:08:49-07:00" level=info msg="executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'web01', pod: 'web01', namespace: 'nginx'"
time="2022-02-16T18:08:49-07:00" level=info msg="starting sniffer cleanup"
time="2022-02-16T18:08:49-07:00" level=info msg="sniffer cleanup completed successfully"

Privileged Mode

time="2022-02-16T18:10:20-07:00" level=info msg="no container specified, taking first container we found in pod."
time="2022-02-16T18:10:20-07:00" level=info msg="selected container: 'web01'"
time="2022-02-16T18:10:20-07:00" level=info msg="sniffing method: privileged pod"
time="2022-02-16T18:10:20-07:00" level=info msg="sniffing on pod: 'web01' [namespace: 'nginx', container: 'web01', filter: '', interface: 'any']"
time="2022-02-16T18:10:20-07:00" level=info msg="creating privileged pod on node: 'ip-192-168-71-207.us-west-2.compute.internal'"
time="2022-02-16T18:10:20-07:00" level=info msg="pod: 'ksniff-2m5ql' created successfully in namespace: 'nginx'"
time="2022-02-16T18:10:20-07:00" level=info msg="waiting for pod successful startup"
time="2022-02-16T18:10:28-07:00" level=info msg="pod: 'ksniff-2m5ql' created successfully on node: 'ip-192-168-71-207.us-west-2.compute.internal'"
time="2022-02-16T18:10:28-07:00" level=info msg="spawning wireshark!"
time="2022-02-16T18:10:28-07:00" level=info msg="starting remote sniffing using privileged pod"
time="2022-02-16T18:10:28-07:00" level=info msg="executing command: '[docker --host unix:///var/run/docker.sock run --rm --name=ksniff-container-pAzcZbOM --net=container:c038f82f09d52f6ad77d3e4c40c08fc8bfaa313bb473c47977c7ccd70ab523d6 maintained/tcpdump -i any -U -w - ]' on container: 'ksniff-privileged', pod: 'ksniff-2m5ql', namespace: 'nginx'"
time="2022-02-16T18:10:28-07:00" level=info msg="starting sniffer cleanup"
time="2022-02-16T18:10:28-07:00" level=info msg="removing privileged container: 'ksniff-privileged'"
time="2022-02-16T18:10:28-07:00" level=info msg="executing command: '[docker --host unix:///var/run/docker.sock rm -f ksniff-container-pAzcZbOM]' on container: 'ksniff-privileged', pod: 'ksniff-2m5ql', namespace: 'nginx'"
time="2022-02-16T18:10:29-07:00" level=info msg="command: '[docker --host unix:///var/run/docker.sock rm -f ksniff-container-pAzcZbOM]' executing successfully exitCode: '0', stdErr :'Error: No such container: ksniff-container-pAzcZbOM\n'"
time="2022-02-16T18:10:29-07:00" level=info msg="privileged container: 'ksniff-privileged' removed successfully"
time="2022-02-16T18:10:29-07:00" level=info msg="removing pod: 'ksniff-2m5ql'"
time="2022-02-16T18:10:29-07:00" level=info msg="removing privileged pod: 'ksniff-2m5ql'"
time="2022-02-16T18:10:29-07:00" level=info msg="privileged pod: 'ksniff-2m5ql' removed"
time="2022-02-16T18:10:29-07:00" level=info msg="pod: 'ksniff-2m5ql' removed successfully"
time="2022-02-16T18:10:29-07:00" level=info msg="sniffer cleanup completed successfully"

[RicHincapie]

Hi all, Are any of you still hitting this issue? I've not been able to reproduce anything similar.

I got the same working with Kubernetes 1.19, Istio 1.9.9: ERRO[0005] failed to start remote sniffing, stopping wireshark error="executing sniffer failed, exit code: '126'", and flag -p worked.

[ZeroDeltaAlpha]

I can get this issue when my nodes try to spin up the pod and Kubernetes set the pod to status "Outofcpu" and then the 126 failure code is due to the pod timing out being created, this may explain some of the above issue. hope this helps someone.