search

iximiuz / cdebug

cdebug - a swiss army knife of container debugging
https://iximiuz.com/en/posts/docker-debug-slim-containers/
Apache License 2.0
1.23k stars 47 forks source link

ln: /proc/1/root/.cdebug-xxxx: Permission denied #1

Closed kappa8219 closed 1 year ago

kappa8219 commented 1 year ago

Got problem running cdebug at several OS-es.

[root@ip-192-168-2-211 cdebug-main]# cat /etc/os-release | grep _NAME
PRETTY_NAME="Amazon Linux 2"

[root@ip-192-168-2-211 cdebug-main]# docker -v
Docker version 20.10.17, build 100c701
[root@ip-192-168-2-211 cdebug-main]# ./cdebug exec -it k8s_POD_monitoring-RESTOFTHENAME
{"status":"Pulling from library/busybox","id":"latest"}
{"status":"Digest: sha256:6bdd92bf5240be1b5f3bf71324f5e371fe59f0e153b27fa1f1620f78ba16963c"}
{"status":"Status: Image is up to date for busybox:latest"}
ln: /proc/1/root/.cdebug-7d23761a: Permission denied

https://github.com/iximiuz/cdebug/blob/ab0d573dc8a210561921212a7f497252bc303d16/cmd/exec/exec.go#L108

Permission to write to proc is here. Symlink can be manually created:


[root@ip-192-168-2-211 cdebug-main]# echo $$
14563
[root@ip-192-168-2-211 cdebug-main]# ln -s /proc/$$/root/bin/ /proc/1/root/.cdebug-1e5b0c0a

Got same mistake at MacOS

iximiuz commented 1 year ago

Nice finding! Could you show the docker ps k8s_POD_monitoring-... and ps auxf outputs from this machine? I haven't tested cdebug with Pods yet but this might be a good opportunity.

kappa8219 commented 1 year ago

Nice finding! Could you show the docker ps k8s_POD_monitoring-... and ps auxf outputs from this machine? I haven't tested cdebug with Pods yet but this might be a good opportunity.

[root@ip-192-168-2-211 cdebug-main]# docker ps | grep k8s_POD_monitoring-node-exporter-prometheus-node-exporter-rprjt_monitoring_3ec71329-0f81-43bf-a775-9d4841f3d734_0
ffb136f899d1   602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5                   "/pause"                 3 weeks ago         Up 3 weeks                   k8s_POD_monitoring-node-exporter-prometheus-node-exporter-rprjt_monitoring_3ec71329-0f81-43bf-a775-9d4841f3d734_0

[root@ip-192-168-2-211 cdebug-main]# ps auxf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S    Oct10   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [rcu_gp]
root         4  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [rcu_par_gp]
root         6  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [kworker/0:0H-ev]
root         8  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [mm_percpu_wq]
root         9  0.0  0.0      0     0 ?        S    Oct10  15:45  \_ [ksoftirqd/0]
root        10  0.0  0.0      0     0 ?        I    Oct10  13:29  \_ [rcu_sched]
root        11  0.0  0.0      0     0 ?        S    Oct10   0:22  \_ [migration/0]
root        13  0.0  0.0      0     0 ?        S    Oct10   0:00  \_ [cpuhp/0]
root        14  0.0  0.0      0     0 ?        S    Oct10   0:00  \_ [cpuhp/1]
root        15  0.0  0.0      0     0 ?        S    Oct10   0:22  \_ [migration/1]
root        16  0.0  0.0      0     0 ?        S    Oct10  16:03  \_ [ksoftirqd/1]
root        18  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [kworker/1:0H-ev]
root        20  0.0  0.0      0     0 ?        S    Oct10   0:00  \_ [kdevtmpfs]
root        21  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [netns]
root        24  0.0  0.0      0     0 ?        S    Oct10   0:13  \_ [kauditd]
root       139  0.0  0.0      0     0 ?        S    Oct10   0:01  \_ [khungtaskd]
root       236  0.0  0.0      0     0 ?        S    Oct10   0:00  \_ [oom_reaper]
root       237  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [writeback]
root       239  0.0  0.0      0     0 ?        S    Oct10   0:01  \_ [kcompactd0]
root       240  0.0  0.0      0     0 ?        SN   Oct10   0:00  \_ [ksmd]
root       241  0.0  0.0      0     0 ?        SN   Oct10   0:08  \_ [khugepaged]
root       296  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [kintegrityd]
root       298  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [kblockd]
root       299  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [blkcg_punt_bio]
root       410  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [tpm_dev_wq]
root       414  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [md]
root       419  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [edac-poller]
root       424  0.0  0.0      0     0 ?        S    Oct10   0:00  \_ [watchdogd]
root       555  0.0  0.0      0     0 ?        S    Oct10   0:15  \_ [kswapd0]
root       678  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [xfsalloc]
root       679  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [xfs_mru_cache]
root       686  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [kthrotld]
root       725  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [nvme-wq]
root       727  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [nvme-reset-wq]
root       729  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [nvme-delete-wq]
root       743  0.0  0.0      0     0 ?        I<   Oct10   0:17  \_ [kworker/1:1H-kb]
root       762  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [ipv6_addrconf]
root       763  0.0  0.0      0     0 ?        I<   Oct10   0:19  \_ [kworker/0:1H-ev]
root       772  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [kstrp]
root      1316  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [xfs-buf/nvme0n1]
root      1317  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [xfs-conv/nvme0n]
root      1318  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [xfs-cil/nvme0n1]
root      1319  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [xfs-reclaim/nvm]
root      1320  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [xfs-eofblocks/n]
root      1321  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [xfs-log/nvme0n1]
root      1322  0.0  0.0      0     0 ?        S    Oct10   4:36  \_ [xfsaild/nvme0n1]
root      1907  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [ena]
root      1931  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [cryptd]
root      2012  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [rpciod]
root      2013  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [kworker/u5:0]
root      2014  0.0  0.0      0     0 ?        I<   Oct10   0:00  \_ [xprtiod]
root      3042  0.0  0.0   4224   768 ?        S    Oct10   6:07  \_ bpfilter_umh
root     23450  0.0  0.0      0     0 ?        S    Oct12   0:20  \_ [jbd2/nvme6n1-8]
root     23451  0.0  0.0      0     0 ?        I<   Oct12   0:00  \_ [ext4-rsv-conver]
root      2709  0.0  0.0      0     0 ?        S    Oct31   0:00  \_ [jbd2/nvme1n1-8]
root      2710  0.0  0.0      0     0 ?        I<   Oct31   0:00  \_ [ext4-rsv-conver]
root      2832  0.0  0.0      0     0 ?        S    Oct31   0:00  \_ [jbd2/nvme2n1-8]
root      2833  0.0  0.0      0     0 ?        I<   Oct31   0:00  \_ [ext4-rsv-conver]
root      3566  0.0  0.0      0     0 ?        S    Oct31   0:02  \_ [jbd2/nvme3n1-8]
root      3567  0.0  0.0      0     0 ?        I<   Oct31   0:00  \_ [ext4-rsv-conver]
root     32598  0.0  0.0      0     0 ?        I    16:17   0:00  \_ [kworker/1:0-cgr]
root      5308  0.0  0.0      0     0 ?        I    16:22   0:00  \_ [kworker/0:13-ev]
root      6602  0.0  0.0      0     0 ?        I    16:59   0:00  \_ [kworker/u4:1-fl]
root      7756  0.0  0.0      0     0 ?        I    17:01   0:00  \_ [kworker/1:3-cgr]
root     10965  0.0  0.0      0     0 ?        I    17:05   0:00  \_ [kworker/u4:0-xf]
root     16069  0.0  0.0      0     0 ?        I    17:10   0:00  \_ [kworker/0:1-eve]
root     27663  0.0  0.0      0     0 ?        I    17:22   0:00  \_ [kworker/0:0-eve]
root         1  0.0  0.0 124180  5472 ?        Ss   Oct10  27:12 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root      1381  0.0  0.1  55764 24168 ?        Ss   Oct10   0:25 /usr/lib/systemd/systemd-journald
root      1401  0.0  0.0  42352  2976 ?        Ss   Oct10   0:04 /usr/lib/systemd/systemd-udevd
root      1402  0.0  0.0 116752  1444 ?        Ss   Oct10   0:00 /usr/sbin/lvmetad -f
root      2018  0.0  0.0  57660  1612 ?        S<sl Oct10   0:26 /sbin/auditd
dbus      2042  0.0  0.0  56320  3424 ?        Ss   Oct10   0:15 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
rpc       2043  0.0  0.0  67276  2332 ?        Ss   Oct10   0:01 /sbin/rpcbind -w
root      2045  0.0  0.0 101988  2328 ?        Ssl  Oct10   0:52 /usr/sbin/irqbalance --foreground
root      2048  0.0  0.0  26492  2896 ?        Ss   Oct10   0:07 /usr/lib/systemd/systemd-logind
rngd      2056  0.0  0.0  94084  3140 ?        Ss   Oct10   0:00 /sbin/rngd -f --fill-watermark=0 --exclude=jitter
chrony    2075  0.0  0.0 118272  2256 ?        S    Oct10   0:28 /usr/sbin/chronyd -F 2
root      2082  0.0  0.0 212000  1908 ?        Ssl  Oct10   0:00 /usr/sbin/gssproxy -D
root      2289  0.0  0.0  98668  3652 ?        Ss   Oct10   0:00 /sbin/dhclient -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid eth0
root      2324  0.0  0.0  98668  3296 ?        Ss   Oct10   0:02 /sbin/dhclient -6 -nw -lf /var/lib/dhclient/dhclient6--eth0.lease -pf /var/run/dhclient6-eth0.pid eth0
root      2486  0.0  0.0  88268  4256 ?        Ss   Oct10   0:03 /usr/libexec/postfix/master -w
postfix   2490  0.0  0.0  88424  5572 ?        S    Oct10   0:00  \_ qmgr -l -t unix -u
postfix  10745  0.0  0.0  88360  6504 ?        S    15:54   0:00  \_ pickup -l -t unix -u
root      2562  0.0  0.1 286168 23172 ?        Ssl  Oct10   1:15 /usr/sbin/rsyslogd -n
root      2563  0.0  0.0 714672  8792 ?        Ssl  Oct10   1:14 /usr/bin/amazon-ssm-agent
root      2638  0.0  0.0 723844 12232 ?        Sl   Oct10   0:53  \_ /usr/bin/ssm-agent-worker
root      2571  0.0  0.0 117160  1248 tty1     Ss+  Oct10   0:00 /sbin/agetty --noclear tty1 linux
root      2572  0.0  0.0 116808  1524 ttyS0    Ss+  Oct10   0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220
root      2618  0.0  0.0 108752  4240 ?        Ss   Oct10   0:00 /usr/sbin/sshd -D
root      2923  0.2  0.2 1599020 36572 ?       Ssl  Oct10  82:00 /usr/bin/containerd
root      3013  1.2  0.3 1978092 63388 ?       Ssl  Oct10 427:24 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root      3194  3.1  0.6 1903440 97712 ?       Ssl  Oct10 1063:49 /usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/kubelet-config.json --kubeconfig /var/lib/kubelet/kub
root      3597  0.0  0.0 712460  6912 ?        Sl   Oct10   5:18 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 1d1790dd2a4281775a14d463e40d58f3cec6d06c2e3072acc3511f5d7addc130 -addres
65535     3721  0.0  0.0    968     4 ?        Ss   Oct10   0:00  \_ /pause
root      3598  0.0  0.0 712460  7332 ?        Sl   Oct10   5:17 /usr/bin/containerd-shim-runc-v2 -namespace moby -id ba77ea236af86cb6699867b3fb8891af59f0bf73ba826a493343e1b7c749589c -addres
65535     3687  0.0  0.0    968     4 ?        Ss   Oct10   0:00  \_ /pause
root      3601  0.0  0.0 712460  7320 ?        Sl   Oct10   2:40 /usr/bin/containerd-shim-runc-v2 -namespace moby -id ffb136f899d1e63532e6cb912dfaaf58921ef131cf5c9e21f2b0b768637d436d -addres
nfsnobo+  3832  0.0  0.0    968     4 ?        Ss   Oct10   0:00  \_ /pause
root      3656  0.0  0.0 712460  7384 ?        Sl   Oct10   1:34 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 5920c57d7c3b925789e2bc0dd6a4d1faa96bf15d8340be3caec52a59ef49baa0 -addres
65535     3782  0.0  0.0    968     4 ?        Ss   Oct10   0:00  \_ /pause
root      3985  0.0  0.0 712460  7336 ?        Sl   Oct10   1:43 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 780aadb2633eab58980493c5bc5e71678b6f45332a2d9779b95b3bbe31737c85 -addres
ec2-user  4006  0.0  0.0 754052 12024 ?        Ssl  Oct10  20:24  \_ /node-termination-handler
root      4090  0.0  0.0 712268  6588 ?        Sl   Oct10   4:23 /usr/bin/containerd-shim-runc-v2 -namespace moby -id d332627e36ca6001ec244f3042a8a86ffb75c9650bfc47895fdb89229b6cb7db -addres
root      4109  0.1  0.2 748884 36256 ?        Ssl  Oct10  53:46  \_ kube-proxy --v=2 --config=/var/lib/kube-proxy-config/config
root      4333  0.0  0.0 712204  6736 ?        Sl   Oct10   4:51 /usr/bin/containerd-shim-runc-v2 -namespace moby -id fc9d7415f06b043920c5cdf3bb96ca38df1a0ed598ff4c92c55ea22431a8888c -addres
nfsnobo+  4356  0.0  0.1 717892 18488 ?        Ssl  Oct10  21:09  \_ /bin/node_exporter --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root --web.listen-address=0.0.0.0:
root      4648  0.0  0.0 712460  9880 ?        Sl   Oct10  32:02 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 5bb9ec9bd75a8d3d70142d9fe3e46dbe6816b4106763aa6de87ebb44f9132115 -addres
root      4670  0.0  0.0  11564   416 ?        Ss   Oct10   0:00  \_ bash /app/entrypoint.sh
root      4720  0.0  0.2 754040 35668 ?        Sl   Oct10  18:39      \_ ./aws-k8s-agent
root      4721  0.0  0.0   4244    64 ?        S    Oct10   0:00      \_ tee -i aws-k8s-agent.log
root      4898  0.0  0.0 712460  7244 ?        Sl   Oct10   3:10 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 49bd7448773854a8a2cdd425cdace18a4c938d01d2568847dc1b6019ad9f3cdd -addres
65535     4992  0.0  0.0    968     4 ?        Ss   Oct10   0:00  \_ /pause
root      4936  0.0  0.0 712204  6788 ?        Sl   Oct10   2:42 /usr/bin/containerd-shim-runc-v2 -namespace moby -id f15a074d45f44f9b713ec059c48c52e7ed938c2392276a1f06b43cab5637b811 -addres
root      5106  0.0  0.0    968     4 ?        Ss   Oct10   0:00  \_ /pause
root      5535  0.0  0.0 712204  7516 ?        Sl   Oct10   4:11 /usr/bin/containerd-shim-runc-v2 -namespace moby -id b3d2b1df6a346d6297dd7f572e223201e0e1a409bc53b1b68e9a6b64da769e7a -addres
root      5643  0.0  0.1 752204 17300 ?        Ssl  Oct10   7:19  \_ /bin/aws-ebs-csi-driver node --endpoint=unix:/csi/csi.sock --logtostderr --v=2
root      5557  0.0  0.0 712268  7224 ?        Sl   Oct10   3:14 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 101d23c5fdd7366924b8169ae727e472632d289f5fd8b23fb4111cbefe0498b4 -addres
root      5604  0.0  0.1 812168 18356 ?        Ssl  Oct10   7:37  \_ /node-problem-detector --logtostderr --config.system-log-monitor=/config/kernel-monitor.json,/config/docker-monitor.json 
root      5833  0.0  0.0 712460  6712 ?        Sl   Oct10   3:38 /usr/bin/containerd-shim-runc-v2 -namespace moby -id b65ddc09bac4e0d625964618f0932dddd8417da1b3acc20b606018d48704d790 -addres
root      5852  0.0  0.0 715152  3128 ?        Ssl  Oct10   0:31  \_ /csi-node-driver-registrar --csi-address=/csi/csi.sock --kubelet-registration-path=/var/lib/kubelet/plugins/ebs.csi.aws.c
root      5950  0.0  0.0 712204  6996 ?        Sl   Oct10   1:44 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 895f67fb8dd8a3aabbde3076ac22a5bae205ed17a26423526c9d46701b4d3d80 -addres
root      5971  0.0  0.0 713804 10852 ?        Ssl  Oct10   5:31  \_ /livenessprobe --csi-address=/csi/csi.sock
root     23463  0.0  0.0 711948  6964 ?        Sl   Oct12   1:43 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 21e1c5d2bdbcc0e903424cc7c128838ee1d9eb9b1529c21d768432702649582f -addres
ec2-user 23504  0.0  0.0    968     4 ?        Ss   Oct12   0:00  \_ /pause
root     23682  0.0  0.0 712204  7040 ?        Sl   Oct12   2:52 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 0b32ca4d9785dc3b519b332ecb21338de61558c7429db289fa92045b38ddf4e7 -addres
ec2-user 23721  0.0  0.0 713920  7700 ?        Ssl  Oct12   1:51  \_ /bin/prometheus-config-reloader --listen-address=:8080 --reload-url=http://127.0.0.1:9090/-/reload --config-file=/etc/pro
root     16536  0.0  0.0 712204  9012 ?        Sl   Oct13   2:43 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 3f7654d5005e4163b9f62413c31f3b3ef07e7c06543c9659c325a4c2e26d0cbf -addres
ec2-user 16583 45.8 49.8 13746328 8014412 ?    Ssl  Oct13 13405:56  \_ /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_librar
root      2739  0.0  0.0 712460  9476 ?        Sl   Oct31   0:23 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 169154c2138290fe42c5f8aa05f44d843016502010fbac73e1daf550703aae32 -addres
10001     2795  0.0  0.0    968     4 ?        Ss   Oct31   0:00  \_ /pause
root      2886  0.0  0.0 712204  9740 ?        Sl   Oct31   0:21 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 791e37794351c90ec79ad350e55cdc6c160d5b52de2cc402ebfcde2ccfbf442a -addres
10001     2929  0.0  0.0    968     4 ?        Ss   Oct31   0:00  \_ /pause
root      3113  0.0  0.0 712460 10132 ?        Sl   Oct31   0:20 /usr/bin/containerd-shim-runc-v2 -namespace moby -id d8b1352aca97b508c1c4aa7b91adbdcdf2b3ca6590fb61ad28c89438f17be474 -addres
10001     3190  0.3  0.3 766000 64248 ?        Ssl  Oct31  11:27  \_ /bin/mimir -target=alertmanager -config.expand-env=true -config.file=/etc/mimir/mimir.yaml
root      3122  0.0  0.0 712204  9492 ?        Sl   Oct31   0:12 /usr/bin/containerd-shim-runc-v2 -namespace moby -id dc8bcad0bf23f72390a92486a0f52cbd78ade8428c6c7afa25a54f4011a89d3f -addres
10001     3158  0.9  0.4 903352 65868 ?        Ssl  Oct31  27:27  \_ /bin/mimir -target=compactor -config.expand-env=true -config.file=/etc/mimir/mimir.yaml
root      3586  0.0  0.0 712460  9432 ?        Sl   Oct31   0:09 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 55f083be0608b7e0cad460749cb9ba4654b5ec177c9efaeba0b20fe7e71eed6c -addres
10001     3665  0.0  0.0    968     4 ?        Ss   Oct31   0:00  \_ /pause
root      3762  0.0  0.0 712204 10512 ?        Sl   Oct31   0:10 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 4a5ba68150642160d81ed552d2115ec862792b2d55c82eed974a1f8e43e23834 -addres
10001     3802  2.0  5.9 3235968 955412 ?      Ssl  Oct31  60:46  \_ /bin/mimir -target=ingester -config.expand-env=true -config.file=/etc/mimir/mimir.yaml -ingester.ring.instance-availabili
root     14251  0.0  0.0 712204  8368 ?        Sl   15:58   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 93b1150e75700c7d78fc385652e4e6e53c065adbc84632b68fdce12194ee963f -addres
65535    14275  0.0  0.0    968     4 ?        Ss   15:58   0:00  \_ /pause
root     14375  0.0  0.0 712460 10092 ?        Sl   15:58   0:01 /usr/bin/containerd-shim-runc-v2 -namespace moby -id ea9394a25dda18a17f21c86d27c847a82449158069094a094bb251b1505228f5 -addres
root     14396  0.0  0.0   4244   724 ?        Ss   15:58   0:00  \_ sleep 14000
root     14554  0.0  0.0  11564  2440 pts/0    Ss   15:58   0:00  \_ sh -c ((clear && bash) || (clear && ash) || (clear && sh))
root     14560  0.0  0.0  11564   220 pts/0    S    15:58   0:00      \_ sh -c ((clear && bash) || (clear && ash) || (clear && sh))
root     14561  0.0  0.0  11564  1872 pts/0    S    15:58   0:00          \_ sh -c ((clear && bash) || (clear && ash) || (clear && sh))
root     14563  0.0  0.0 122260  3476 pts/0    S    15:58   0:00              \_ bash
root     29565  0.0  0.0 160372  4092 pts/0    R+   17:24   0:00                  \_ ps auxf
root     15293  0.0  0.0 133004  3104 ?        Ss   15:59   0:00 /usr/sbin/crond -n
kappa8219 commented 1 year ago

BTW, on this node example works fine:

[root@ip-192-168-2-211 cdebug-main]# docker run -d --rm \
>   --name my-distroless gcr.io/distroless/nodejs \
>   -e 'setTimeout(() => console.log("Done"), 99999999)'
Unable to find image 'gcr.io/distroless/nodejs:latest' locally
latest: Pulling from distroless/nodejs
8fdb1fc20e24: Pull complete 
fda4ba87f6fb: Pull complete 
a1f1879bb7de: Pull complete 
e0f01260a67b: Pull complete 
Digest: sha256:73b9873c8d382695b39649a0afea9fd80f77b17cdfc25dc09dbc7a1bea7518f3
Status: Downloaded newer image for gcr.io/distroless/nodejs:latest
a70b1b9afb982ef23c267ae1b1b672892ed04aebe4a3d28e4b0d88b20ada5155
[root@ip-192-168-2-211 cdebug-main]# 
[root@ip-192-168-2-211 cdebug-main]# 
[root@ip-192-168-2-211 cdebug-main]# 
[root@ip-192-168-2-211 cdebug-main]# cdebug exec -it my-distroless
bash: cdebug: command not found
[root@ip-192-168-2-211 cdebug-main]# ./cdebug exec -it my-distroless
{"status":"Pulling from library/busybox","id":"latest"}
{"status":"Digest: sha256:6bdd92bf5240be1b5f3bf71324f5e371fe59f0e153b27fa1f1620f78ba16963c"}
{"status":"Status: Image is up to date for busybox:latest"}
/ #
kappa8219 commented 1 year ago

Looks like something is wrong with this pause-image:

[root@ip-192-168-2-211 cdebug-main]# docker run -d --rm   --name my-pause 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5   -e 'setTimeout(() => console.log("Done"), 99999999)'
b196f9ed88d7ca7bddcf7093f1eb608fda109a3565837ece11accf519ef8567a
[root@ip-192-168-2-211 cdebug-main]# ./cdebug exec -it my-pause
{"status":"Pulling from library/busybox","id":"latest"}
{"status":"Digest: sha256:6bdd92bf5240be1b5f3bf71324f5e371fe59f0e153b27fa1f1620f78ba16963c"}
{"status":"Status: Image is up to date for busybox:latest"}
ln: /proc/1/root/.cdebug-cd3d032a: Permission denied
kappa8219 commented 1 year ago

Trying to compare. (quite funny history of distroless)

[root@ip-192-168-2-211 cdebug-main]# docker history 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5
IMAGE          CREATED         CREATED BY                                    SIZE      COMMENT
6996f8da07bd   17 months ago   ENTRYPOINT ["/pause"]                         0B        buildkit.dockerfile.v0
<missing>      17 months ago   USER 65535:65535                              0B        buildkit.dockerfile.v0
<missing>      17 months ago   ADD bin/pause-linux-amd64 /pause # buildkit   683kB     buildkit.dockerfile.v0
<missing>      17 months ago   ARG ARCH                                      0B        buildkit.dockerfile.v0

[root@ip-192-168-2-211 cdebug-main]# docker history gcr.io/distroless/nodejs:latest
IMAGE          CREATED        CREATED BY        SIZE      COMMENT
cc408ea903ad   52 years ago   bazel build ...   137MB     
<missing>      52 years ago   bazel build ...   2.34MB    
<missing>      52 years ago   bazel build ...   18MB      
<missing>      52 years ago   bazel build ...   2.34MB
kappa8219 commented 1 year ago

Looks like cdebug works only for images with root user(default when no USER specified in Dockerfile).

I'v succeded with java distroless one:

[root@ip-192-168-3-74 cdebug-main]# docker history xxx.dkr.ecr.us-east-1.amazonaws.com/eureka-service:xxx
IMAGE          CREATED        CREATED BY               SIZE      COMMENT
68a4bcb5b4b1   2 years ago    jib-maven-plugin:1.8.0   925B      classes
<missing>      2 years ago    jib-maven-plugin:1.8.0   482B      resources
<missing>      2 years ago    jib-maven-plugin:1.8.0   49.2MB    dependencies
<missing>      52 years ago   bazel build ...          170MB     
<missing>      52 years ago   bazel build ...          8.41MB    
<missing>      52 years ago   bazel build ...          1.93MB    
<missing>      52 years ago   bazel build ...          15.1MB    
<missing>      52 years ago   bazel build ...          1.79MB

Thanks for the usefull tool!

iximiuz commented 1 year ago

Finally got a chance to debug it myself. The "issue" is caused by the missing privileges. Even when both, the target and the debugger container, use the root user, it might happen that the target runs in the privileged mode while the debugger by default is not privileged. You can work around it with cdebug exec --privileged -it <target>. I'm also adding an F.A.Q section to the README to address this and other potential questions.

I'll also explore a potential improvement for the cdebug exec command - it might be possible to detect if the target container runs in the privileged mode and use the privileged debugger container automatically.

Great discovery! Thanks for reporting and for thorough debugging!

iximiuz commented 1 year ago

Here is the improvement. Thanks for helping it happen!