netns return empty string so nothing is ever captured

[seb54000]

Hi, I'm really interested into testing ksniff. I just install it on an AWS standard AMI with a local microk8s cluster installed along with docker as container runtime.

I have a simple webserver where I launch ksniff on. But there is never a single packet coming in wireshark (or stdout when I launch just ksniff with -o -). Of course I make in parralel plenty of HTTP request to the simple webserver "bgd"

After issuing a ps command, I can see how the ksniff container command has been run and I thinks this may be the cause of my problem but I don't know how to specify a net (like net=host ? but will it work ?)

# Command was 
kubectl sniff -n default bgd-xxx-yyy 

INFO[0005] executing command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/run/containerd/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
    export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
    crictl pull docker.io/maintained/tcpdump:latest >/dev/null
    netns=$(crictl inspect 5a0038d78596179fa055aa1395ae515f13011e06751b2271131a07f5a87ba27f | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
    exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-sluEcVnE tcpdump -i any -U -w -  
    ]' on container: 'ksniff-privileged', pod: 'ksniff-t4r9t', namespace: 'default' 

# ps gives
 4613 root      0:00 ctr -a /run/containerd/containerd.sock run --rm --with-ns network: docker.io/maintained/tcpdump:latest ksniff-container-sluEcVnE tcpdump -i any -U -w -
 4670 root      0:00 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id ksniff-container-sluEcVnE -address /run/containerd/containerd.sock

Notice the --with-ns network:

Any help is welcome and I can of course give more details, please ask

The "app" I'm using for this test (image: quay.io/redhatworkshops/bgd:latest) and yaml is :

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: bgd
  name: bgd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bgd
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: bgd
    spec:
      containers:
      - image: quay.io/redhatworkshops/bgd:latest
        name: bgd
        env:
        - name: COLOR
          value: "green"
        resources: {}

[seb54000]

Still searching from my side, I just see that despite I'm installing docker runtime through yum on the AMI, it also installs containerd. SO microk8s seems to use containerd and this is why ksniff uses the containerd config. Mays this be linked with my problem ?

================================================================================
 Package        Arch       Version                  Repository             Size
================================================================================
Installing:
 docker         x86_64     20.10.17-1.amzn2         amzn2extra-docker      39 M
Installing for dependencies:
 containerd     x86_64     1.6.6-1.amzn2            amzn2extra-docker      27 M
 pigz           x86_64     2.3.4-1.amzn2.0.1        amzn2-core             81 k
 runc           x86_64     1.1.3-1.amzn2            amzn2extra-docker     2.9 M

[seb54000]

I'm really not sure of what container runtime microk8s is using... https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/#:~:text=If%20the%20%2D%2Dcontainer%2Druntime%2Dendpoint%20flag%20is%20present%2C,sock%20is%20the%20containerd%20endpoint.

sudo tr \\0 ' ' < /proc/"$(pgrep kubelet)"/cmdline

BOOT_IMAGE=/boot/vmlinuz-4.14.268-205.500.amzn2.x86_64 root=UUID=f22063bb-28ab-42bc-ba62-b9130b861ee7 ro console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0 LANG=en_US.UTF-8 KEYTABLE=us

According to the link: if the --container-runtime flag is not remote, you use the dockershim socket with Docker Engine

kubectl version

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.3-2+63243a96d1c393", GitCommit:"63243a96d1c3932c704cf0625f5834411bc3b2e6", GitTreeState:"clean", BuildDate:"2022-07-19T17:57:14Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

[seb54000]

I just try to switch microk8s to docker for container-runtime (https://stackoverflow.com/questions/61119975/is-it-possible-to-set-microk8s-to-use-docker-engine-instead-of-containerd ) but it doesn't work, the cluster do not launch any pods and no events

Kubelet logs are clearer after checking (sorry) août 26 08:13:51 ip-172-31-16-109.ec2.internal microk8s.daemon-kubelite[3913]: Error: failed to validate kubelet flags: unsupported CRI runtime: "docker", only "remote" is curre

[seb54000]

For information, kubelet conf is as following

sudo cat /var/snap/microk8s/current/args/kubelet

--kubeconfig=${SNAP_DATA}/credentials/kubelet.config
--cert-dir=${SNAP_DATA}/certs
--client-ca-file=${SNAP_DATA}/certs/ca.crt
--anonymous-auth=false
--root-dir=${SNAP_COMMON}/var/lib/kubelet
--fail-swap-on=false
--feature-gates=DevicePlugins=true
--eviction-hard="memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi"
--container-runtime=remote
--container-runtime-endpoint=${SNAP_COMMON}/run/containerd.sock
--containerd=${SNAP_COMMON}/run/containerd.sock
--node-labels="microk8s.io/cluster=true,node.kubernetes.io/microk8s-controlplane=microk8s-controlplane"
--authentication-token-webhook=true
--read-only-port=0
--cluster-domain=cluster.local
--cluster-dns=10.152.183.10

[seb54000]

With help of ps, I guess there is a mismatch between containerd socket used by microk8s and the one (default containerd) where ksniff is inspecting the containers to get network infos

ps -aux |grep containerd
root     12852  0.0  0.8 1445188 36012 ?       Ssl  08:02   0:00 /usr/bin/containerd
root     12898  0.0  1.6 1361004 64944 ?       Ssl  08:02   0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --default-ulimit nofile=32768:65536
root     19418  6.6  1.1 1912688 47836 ?       Ssl  08:21   0:20 /snap/microk8s/3597/bin/containerd --config /var/snap/microk8s/3597/args/containerd.toml --root /var/snap/microk8s/common/var/lib/containerd --state /var/snap/microk8s/common/run/containerd --address /var/snap/microk8s/common/run/containerd.sock
root     20604  0.0  0.2 710504  8940 ?        Sl   08:21   0:00 /var/lib/snapd/snap/microk8s/3597/bin/containerd-shim-runc-v1 -namespace k8s.io -id 2f4b67f8277aa44692ef96c691fd9bfd3186f90b91b28557c9614fec0173f659 -address /var/snap/microk8s/common/run/containerd.sock
root     21063  0.0  0.2 710504  8448 ?        Sl   08:21   0:00 /var/lib/snapd/snap/microk8s/3597/bin/containerd-shim-runc-v1 -namespace k8s.io -id e02a1a73114c7c32cb302d91d0fba8f11c83a3d4dbf912e329d8078f2b4a1d8f -address /var/snap/microk8s/common/run/containerd.sock
...

--address /var/snap/microk8s/common/run/containerd.sock

[seb54000]

When I use the ctr command from microk8s, I can have kind of inspect but the jq command is not working. The problem is : how to give ksniff the info about the contaierd sock to use : seems related to : https://github.com/eldadru/ksniff/issues/65

sudo /snap/bin/microk8s.ctr container info fbc451b82bdfbef2fd91c3fa3625448ee6690228dec033cde56f7c4b2920776f

sudo /snap/bin/microk8s.ctr container info fbc451b82bdfbef2fd91c3fa3625448ee6690228dec033cde56f7c4b2920776f | jq .info
null

[seb54000]

Grrr, I don't understand

kubectl sniff -n default bgd-7b56ddcf4b-jnvzb -p --socket /var/snap/microk8s/common/run/containerd.sock
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'bgd'
INFO[0000] sniffing method: privileged pod
INFO[0000] sniffing on pod: 'bgd-7b56ddcf4b-jnvzb' [namespace: 'default', container: 'bgd', filter: '', interface: 'any']
INFO[0000] creating privileged pod on node: 'ip-172-31-16-109.ec2.internal'
INFO[0000] pod: 'ksniff-jc7pp' created successfully in namespace: 'default'
INFO[0000] waiting for pod successful startup
INFO[0005] pod: 'ksniff-jc7pp' created successfully on node: 'ip-172-31-16-109.ec2.internal'
INFO[0005] spawning wireshark!
INFO[0005] starting remote sniffing using privileged pod
INFO[0005] executing command: '[/bin/sh -c
    set -ex
    export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
    export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
    crictl pull docker.io/maintained/tcpdump:latest >/dev/null
    netns=$(crictl inspect 6b5246c84058b072cb4f2a22208543107ac2a0074124faa2634a5886b4c5ecc9 | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
    exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-YIVNUQiY tcpdump -i any -U -w -
    ]' on container: 'ksniff-privileged', pod: 'ksniff-jc7pp', namespace: 'default'
INFO[0005] starting sniffer cleanup
INFO[0005] removing privileged container: 'ksniff-privileged'
INFO[0005] executing command: '[/bin/sh -c
    set -ex
    export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_ID="ksniff-container-YIVNUQiY"
    chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
    ]' on container: 'ksniff-privileged', pod: 'ksniff-jc7pp', namespace: 'default'
INFO[0005] command: '[/bin/sh -c
    set -ex
    export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_ID="ksniff-container-YIVNUQiY"
    chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
    ]' executing successfully exitCode: '1', stdErr :'+ export 'CONTAINERD_SOCKET=/var/snap/microk8s/common/run/containerd.sock'
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_ID=ksniff-container-YIVNUQiY'
+ chroot /host ctr -a /var/snap/microk8s/common/run/containerd.sock task kill -s SIGKILL ksniff-container-YIVNUQiY
ctr: container "ksniff-container-YIVNUQiY" in namespace "k8s.io": not found
'
INFO[0005] privileged container: 'ksniff-privileged' removed successfully
INFO[0005] removing pod: 'ksniff-jc7pp'
INFO[0005] removing privileged pod: 'ksniff-jc7pp'
INFO[0005] privileged pod: 'ksniff-jc7pp' removed
INFO[0005] pod: 'ksniff-jc7pp' removed successfully
INFO[0005] sniffer cleanup completed successfully
Error: exit status 1

container "ksniff-container-YIVNUQiY" in namespace "k8s.io": not found

But...

sudo /snap/bin/microk8s.ctr namespaces ls
NAME   LABELS
k8s.io

sudo /snap/bin/microk8s.ctr container info 6b5246c84058b072cb4f2a22208543107ac2a0074124faa2634a5886b4c5ecc9 | jq .Image
"quay.io/redhatworkshops/bgd:latest"

sudo /snap/bin/microk8s.ctr container ls | grep 6b5246c84058b072cb4f2a22208543107ac2a0074124faa2634a5886b4c5ecc9
6b5246c84058b072cb4f2a22208543107ac2a0074124faa2634a5886b4c5ecc9    quay.io/redhatworkshops/bgd:latest             io.containerd.runc.v1

[seb54000]

Hello, still searching if you have any ideas. I did the whole test https://github.com/eldadru/ksniff/issues/65#issuecomment-984790977 reffered by @skyhisi including wrapper for /usr/bin/ctr (I made a backup of the ctr binary)

Thi works a bit better (wireshark is lauching and the right network seems to be found) but an another error occur : ctr: failed to create temp dir: stat /run/user/0/snap.microk8s: no such file or directory

The whole log is :

kubectl sniff -n default bgd-7b56ddcf4b-hpxr7 -p --socket /var/snap/microk8s/common/run/containerd.sock
INFO[0000] no container specified, taking first container we found in pod. 
INFO[0000] selected container: 'bgd'                    
INFO[0000] sniffing method: privileged pod              
INFO[0000] sniffing on pod: 'bgd-7b56ddcf4b-hpxr7' [namespace: 'default', container: 'bgd', filter: '', interface: 'any'] 
INFO[0000] creating privileged pod on node: 'ip-172-31-20-62.ec2.internal' 
INFO[0000] pod: 'ksniff-8sbzx' created successfully in namespace: 'default' 
INFO[0000] waiting for pod successful startup           
INFO[0003] pod: 'ksniff-8sbzx' created successfully on node: 'ip-172-31-20-62.ec2.internal' 
INFO[0003] spawning wireshark!                          
INFO[0003] starting remote sniffing using privileged pod 
INFO[0003] executing command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
    export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
    crictl pull docker.io/maintained/tcpdump:latest >/dev/null
    netns=$(crictl inspect a5023573788d6a67593ad80567c486337646e100db8d071d70535c17774f58ec | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
    exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-RIvqkpPo tcpdump -i any -U -w -  
    ]' on container: 'ksniff-privileged', pod: 'ksniff-8sbzx', namespace: 'default' 
INFO[0004] command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_RUNTIME_ENDPOINT="unix:///host${CONTAINERD_SOCKET}"
    export IMAGE_SERVICE_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT}
    crictl pull docker.io/maintained/tcpdump:latest >/dev/null
    netns=$(crictl inspect a5023573788d6a67593ad80567c486337646e100db8d071d70535c17774f58ec | jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path' | tr -d '"')
    exec chroot /host ctr -a ${CONTAINERD_SOCKET} run --rm --with-ns "network:${netns}" docker.io/maintained/tcpdump:latest ksniff-container-RIvqkpPo tcpdump -i any -U -w -  
    ]' executing successfully exitCode: '1', stdErr :'+ export 'CONTAINERD_SOCKET=/var/snap/microk8s/common/run/containerd.sock'
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_RUNTIME_ENDPOINT=unix:///host/var/snap/microk8s/common/run/containerd.sock'
+ export 'IMAGE_SERVICE_ENDPOINT=unix:///host/var/snap/microk8s/common/run/containerd.sock'
+ crictl pull docker.io/maintained/tcpdump:latest
+ tr -d '"'
+ jq '.info.runtimeSpec.linux.namespaces[] | select(.type == "network") | .path'
+ crictl inspect a5023573788d6a67593ad80567c486337646e100db8d071d70535c17774f58ec
+ netns=/proc/1591/ns/net
+ exec chroot /host ctr -a /var/snap/microk8s/common/run/containerd.sock run --rm --with-ns network:/proc/1591/ns/net docker.io/maintained/tcpdump:latest ksniff-container-RIvqkpPo tcpdump -i any -U -w -
ctr: failed to create temp dir: stat /run/user/0/snap.microk8s: no such file or directory
' 

### Here it hangs while wireshark is launched (trying desperatly to show some capture packets that never comes
### Following lines appears when I close wireshark

INFO[0004] remote sniffing using privileged pod completed 
INFO[0022] starting sniffer cleanup                     
INFO[0022] removing privileged container: 'ksniff-privileged' 
INFO[0022] executing command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_ID="ksniff-container-RIvqkpPo"
    chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
    ]' on container: 'ksniff-privileged', pod: 'ksniff-8sbzx', namespace: 'default' 
INFO[0022] command: '[/bin/sh -c 
    set -ex
    export CONTAINERD_SOCKET="/var/snap/microk8s/common/run/containerd.sock"
    export CONTAINERD_NAMESPACE="k8s.io"
    export CONTAINER_ID="ksniff-container-RIvqkpPo"
    chroot /host ctr -a ${CONTAINERD_SOCKET} task kill -s SIGKILL ${CONTAINER_ID}
    ]' executing successfully exitCode: '1', stdErr :'+ export 'CONTAINERD_SOCKET=/var/snap/microk8s/common/run/containerd.sock'
+ export 'CONTAINERD_NAMESPACE=k8s.io'
+ export 'CONTAINER_ID=ksniff-container-RIvqkpPo'
+ chroot /host ctr -a /var/snap/microk8s/common/run/containerd.sock task kill -s SIGKILL ksniff-container-RIvqkpPo
ctr: container "ksniff-container-RIvqkpPo" in namespace "k8s.io": not found
' 
INFO[0022] privileged container: 'ksniff-privileged' removed successfully 
INFO[0022] removing pod: 'ksniff-8sbzx'                 
INFO[0022] removing privileged pod: 'ksniff-8sbzx'      
INFO[0022] privileged pod: 'ksniff-8sbzx' removed       
INFO[0022] pod: 'ksniff-8sbzx' removed successfully     
INFO[0022] sniffer cleanup completed successfully       

[bostrt]

hey @seb54000 i just want to acknowledge your issue. I haven't dug into yet but I plan to soon. The microk8s bridge for ksniff is pretty complicated so there's a lot to look at, as you have!

[seb54000]

Thanks @bostrt , no hurry here, it is just for testing. When we will figure out how to cope with my microk8s config I will be happy to contribute a README in the repo for this "specific" use case.