RFE: trace TLS in container

[vincentmli]

Hi,

for example I have a netshoot pod running in kubernetes, when I run curl

kubectl exec -it netshoot-hostnetwork  -- curl -k -v https://10.1.34.88/

I want to trace the TLS connection from the curl from the netshoot-hostnetwork pod, the curl in netshoot-hostnetwork pod has libssl below in the pod namespace

kubectl exec -it netshoot-hostnetwork  -- ldd /usr/bin/curl
    /lib/ld-musl-x86_64.so.1 (0x7f932eda0000)
    libcurl.so.4 => /usr/lib/libcurl.so.4 (0x7f932ece5000)
    libz.so.1 => /lib/libz.so.1 (0x7f932eccb000)
    libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f932eda0000)
    libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x7f932eca6000)
    libssl.so.1.1 => /lib/libssl.so.1.1 (0x7f932ec25000)
    libcrypto.so.1.1 => /lib/libcrypto.so.1.1 (0x7f932e9a4000)
    libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x7f932e998000)
    libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x7f932e975000)

is this possible? or in general, could we improve ecapture to capture container TLS traffic?

[vincentmli]

I could do this:

find the netshoot docker ID

docker ps | grep -w netshoot
b37ffd7a8341   nicolaka/netshoot           "/bin/sleep 3600"        50 minutes ago      Up 50 minutes                k8s_netshoot_netshoot-hostnetwork_default_751666e9-5a23-4da1-952f-379288b47f97_0

docker inspect the ID

docker inspect b37ffd7a8341  | grep '"MergedDir"'
                "MergedDir": "/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged",

find libssl

find /var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged -name "libssl*"

/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1
/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/usr/lib/libssl.so.1.1

use ecapture with the correct libssl path

ecapture tls --libssl="/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1" --hex
2022/05/30 14:02:27 pid info :2825032
2022/05/30 14:02:27 start to run EBPFProbeOPENSSL module
2022/05/30 14:02:27 start to run EBPFProbeGNUTLS module
2022/05/30 14:02:27 HOOK type:2, binrayPath:/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1
2022/05/30 14:02:27 libPthread so Path:/lib64/libpthread.so.0
2022/05/30 14:02:27 target all process. 
2022/05/30 14:02:27 start to run EBPFProbeNSPR module
2022/05/30 14:02:27 stat /usr/lib/libnspr4.so: no such file or directory
2022/05/30 14:02:27 HOOK type:2, binrayPath:/lib64/libgnutls.so.30
2022/05/30 14:02:27 target all process. 

execute the curl from netshoot pod

kubectl exec -it netshoot-hostnetwork  -- curl -k -v https://10.1.34.88/
*   Trying 10.1.34.88:443...
* Connected to 10.1.34.88 (10.1.34.88) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=US; ST=WA; L=Seattle; O=MyCompany; OU=IT; CN=localhost.localdomain; emailAddress=root@localhost.localdomain
*  start date: Oct  1 19:29:04 2020 GMT
*  expire date: Sep 29 19:29:04 2030 GMT
*  issuer: C=US; ST=WA; L=Seattle; O=MyCompany; OU=IT; CN=localhost.localdomain; emailAddress=root@localhost.localdomain
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET / HTTP/1.1
> Host: 10.1.34.88
> User-Agent: curl/7.83.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: BigIP
* HTTP/1.0 connection set to keep alive
< Connection: Keep-Alive
< Content-Length: 10
< 
* Connection #0 to host 10.1.34.88 left intact
 IT WORKS

the ecapture output:

2022/05/30 14:03:34 PID:2825502, Comm:curl, TID:2825502, Version:TLS1_2_VERSION, Send 74 bytes to [ADDR_NOT_FOUND], Payload:
GET / HTTP/1.1
Host: 10.1.34.88
User-Agent: curl/7.83.1
Accept: */*

2022/05/30 14:03:34 PID:2825502, Comm:curl, TID:2825502, Version:TLS1_2_VERSION, Recived 88 bytes from [ADDR_NOT_FOUND], Payload:
HTTP/1.0 200 OK
Server: BigIP
Connection: Keep-Alive
Content-Length: 10

 IT WORKS 

[cfc4n]

yes, you are right. eh....Is your problem solved?

[vincentmli]

yes it is resolved, thanks for this great project :)

[BurlyLuo]

yes it is resolved, thanks for this great project :)

Thanks the test. Can we have a sample to use this tool into the container?

[cfc4n]

@BurlyLuo That is a good idea. can you send a PR for it? or an article?

[vincentmli]

hm, this brings me thoughts about multi tenant cloud environment, that someone could run ecapture as container and gain privilege on the node and sniff other tenant containers TLS connection, is that possible ?

[cfc4n]

Of course, if the container is authorized with the SYS_ADMIN permission, then it can obtain the communication plaintext of all networks on this host.

So, That is an other topic about eBPF security on runtime.

ref: https://github.com/ehids/ebpf-slide/blob/master/security/us-21-With-Friends-Like-EBPF-Who-Needs-Enemies.pdf demo: https://www.youtube.com/watch?v=zgsHc8apFGs posts on chinese: https://www.cnxct.com/evil-use-ebpf-and-how-to-detect-ebpf-rootkit-in-linux/?f=ecapture-github

[vincentmli]

I wonder if https://github.com/cilium/tetragon is able to detect this scenario, I assume your ehids could detect this too

[cfc4n]

tetragon is the learning objective of ehids.

[vincentmli]

I am not familiar with docker, can we create a docker file for ecapture to run ecapture in container/pod ?

[cfc4n]

I think It is not necessary. via: https://github.com/ehids/ecapture/pull/23

[vincentmli]

ok, good, just one more question, is it possible to give --libssl argument multiple libssl path so ecapture could capture multiple TLS connections for multiple processes/commands ? I guess we can run multiple ecapture commands to achieve that, just wonder :)

[cfc4n]

1, eCapture can capture all process TLS paintext who use the same libssl.so default. and can use pid arg to filter. 2, Can run multiple eCapture with every libssl path to achieve that. 3, Of course, an eCapture process can hook multiple ssl/tls SO file, but I don't think this is a necessary requirement.

[vincentmli]

ah, right I got 1), I am thinking in k8s pod/container scenario that each pod/container might have their own libssl copy, yes 2) can achieve that.

[vincentmli]

by the way, I recorded a short video playing with ecapture https://youtu.be/Au1YeB0nz3g

[cfc4n]

Good job. I'll create User Manual WIKI tomorrow with your video, thanks. 😊

[vincentmli]

Hi @cfc4n sorry to bother you again, I added the ecapture binary in netshoot pod like https://github.com/vincentmli/netshoot/commit/72633ba1da40852fbe32874634eebdfa5bf8e625 so I can run ecapture from netshoot pod in k8s, the netshoot pod yaml file has privilege permission like below

apiVersion: v1
kind: Pod
metadata:
 name: netshoot-ecap
spec:
 hostPID: true
 hostNetwork: true
 nodeSelector:
    dedicated: master
 containers:
   - name: netshoot-ecap
     image: vli39/netshoot:ecap
     command:
       - /bin/sleep
       - "3600"
     volumeMounts:
       - mountPath: /mnt
         name: host-slash
     securityContext:
       privileged: true
 volumes:
   - name: host-slash
     hostPath:
       path: /
       type: ''

you can see I mounted the node root / in netshoot pod /mnt to access the host libssl, but it errors out with "lstat /etc/ld.so.conf: no such file or directory", I wonder why ecapture is trying to check host /etc/ld.so.conf, ecapture is statically build, I suspect the host libssl requiring host /etc/ld.so.conf, but I can't mount host /etc to netshoot pod /etc, wonder if you have better idea :)

kubectl exec -it netshoot-ecap -- /bin/bash
bash-5.1# ls -l /mnt/lib64/libssl*
lrwxrwxrwx    1 root     root            16 Mar 30  2021 /mnt/lib64/libssl.so -> libssl.so.1.1.1g
lrwxrwxrwx    1 root     root            16 Mar 30  2021 /mnt/lib64/libssl.so.1.1 -> libssl.so.1.1.1g
-rwxr-xr-x    1 root     root        615576 Mar 30  2021 /mnt/lib64/libssl.so.1.1.1g
-rwxr-xr-x    1 root     root        394632 Dec 18  2020 /mnt/lib64/libssl3.so

bash-5.1# ecapture tls --libssl="/mnt/lib64/libssl.so.1.1.1g" --hex
2022/06/03 14:51:29 pid info :1391071
2022/06/03 14:51:29 start to run EBPFProbeOPENSSL module
2022/06/03 14:51:29 lstat /etc/ld.so.conf: no such file or directory
2022/06/03 14:51:29 invalid argument

[cfc4n]

eh, eCapture will find pthread.so to capture IP ADDRESS default , and you can set pthread.so argument with pthread falg.

[vincentmli]

ecapture is statically built

ldd /usr/local/bin/ecapture
    not a dynamic executable

but are you saying eCapture dynamically linked with pthread.so? how do I specify ecapture with pthread flag? sorry I am not following you :)

[BurlyLuo]

eh, eCapture will find pthread.so to capture IP ADDRESS default , and you can set pthread.so argument with pthread falg.

bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libssl.so.1.1" --pthread="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libpthread.so.0" 
2022/06/05 04:36:54 pid info :10668
2022/06/05 04:36:54 start to run EBPFProbeOPENSSL module
2022/06/05 04:36:54 start to run EBPFProbeGNUTLS module
2022/06/05 04:36:54 invalid argument
bash-5.1# 

bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libssl.so.1.1" --pthread="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libpthread.so.0"  -h
NAME:
        tls - alias name:openssl , use to capture tls/ssl text content without CA cert.

USAGE:
        ecapture tls [flags]

DESCRIPTION:
        use eBPF uprobe to capture process event data, not used libpcap.
        Can used to trace, debug, database audit, security event aduit etc.

OPTIONS:
      --curl=""         curl or wget file path, use to dectet openssl.so path, default:/usr/bin/curl
      --firefox=""      firefox file path, default: /usr/lib/firefox/firefox.
      --gnutls=""       libgnutls.so file path, will automatically find it from curl default.
  -h, --help[=false]    help for tls
      --libssl=""       libssl.so file path, will automatically find it from curl default.
      --nspr=""         libnspr44.so file path, will automatically find it from curl default.
      --pthread=""      libpthread.so file path, use to hook connect to capture socket FD.will automatically find it from curl.
      --wget=""         wget file path, default: /usr/bin/wget.

GLOBAL OPTIONS:
  -d, --debug[=false]   enable debug logging
      --hex[=false]     print byte strings as hex encoded strings
  -p, --pid=0           if pid is 0 then we target all pids

bash-5.1# 

[cfc4n]

ecapture is statically built

ldd /usr/local/bin/ecapture
  not a dynamic executable

but are you saying eCapture dynamically linked with pthread.so? how do I specify ecapture with pthread flag? sorry I am not following you :)

sure, ecapture is statically built . but eCapture hook connect()function to capture IP ADDRESS ,sometimes connect function was define inpthread.so. so need to search the path.

https://github.com/ehids/ecapture/blob/ed5ebf13bb12873292625d550ffa902c96124388/kern/openssl_kern.c#L285-L290

https://github.com/ehids/ecapture/blob/ed5ebf13bb12873292625d550ffa902c96124388/user/probe_openssl.go#L157-L161

[cfc4n]

sometimes , defined in /lib64/libc.so.6 , you can use lddwhich curl|grep libc.so|awk '{print $3}'|xargs objdump -T |grep connect to confirm it.

[root@VM-16-13-centos bin]# ldd `which curl`|grep libc.so|awk '{print $3}'|xargs objdump -T |grep connect
0000000000112f90  w   DF .text  000000000000009b  GLIBC_2.2.5 __connect
0000000000112f90  w   DF .text  000000000000009b  GLIBC_2.2.5 connect

@BurlyLuo @vincentmli It will skip directory search if lib path args was set.

[vincentmli]

ah, sorry I missed the output libPthread so Path:/lib64/libpthread.so.0 when run ecapture normally

 ecapture tls  --libssl=/lib64/libssl.so.1.1.1g
2022/06/05 10:53:59 pid info :3232826
2022/06/05 10:53:59 start to run EBPFProbeOPENSSL module
2022/06/05 10:53:59 start to run EBPFProbeGNUTLS module
2022/06/05 10:53:59 stat /usr/lib/libgnutls.so.30: no such file or directory
2022/06/05 10:53:59 start to run EBPFProbeNSPR module
2022/06/05 10:53:59 stat /usr/lib/libnspr4.so: no such file or directory
2022/06/05 10:53:59 HOOK type:2, binrayPath:/lib64/libssl.so.1.1.1g
2022/06/05 10:53:59 libPthread so Path:/lib64/libpthread.so.0 <====MISSED THIS 
2022/06/05 10:53:59 target all process. 

but still error when I run ecapture in container with --pthread=/mnt/lib64/libpthread.so.0, it looks to me when putting ecapture in container, for some reason, ecapture could not find the correct pthread lib even with --pthread argument, and still try to locate /etc/ld.so.conf

kubectl exec -it netshoot-ecap -- /bin/bash
bash-5.1# 

bash-5.1# ls -l /mnt/lib64/libpth*
-rwxr-xr-x    1 root     root        320504 Jul 20  2020 /mnt/lib64/libpthread-2.28.so
lrwxrwxrwx    1 root     root            27 Jul 20  2020 /mnt/lib64/libpthread.so -> ../../lib64/libpthread.so.0
lrwxrwxrwx    1 root     root            18 Jul 20  2020 /mnt/lib64/libpthread.so.0 -> libpthread-2.28.so

bash-5.1# ecapture tls --libssl=/mnt/lib64/libssl.so.1.1.1g --pthread=/mnt/lib64/libpthread.so.0
2022/06/05 15:01:40 pid info :3237812
2022/06/05 15:01:40 start to run EBPFProbeOPENSSL module
2022/06/05 15:01:40 start to run EBPFProbeGNUTLS module
2022/06/05 15:01:40 lstat /etc/ld.so.conf: no such file or directory
2022/06/05 15:01:40 invalid argument

[vincentmli]

tried create /etc/ld.so.conf with mounted host / in container, still can't find connect I guess

bash-5.1# echo "/mnt/lib64" > /etc/ld.so.conf
bash-5.1# cat /etc/ld.so.conf
/mnt/lib64

bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libssl.so.1.0.2k"  --hex
2022/06/05 15:32:04 pid info :3257576
2022/06/05 15:32:04 start to run EBPFProbeOPENSSL module
2022/06/05 15:32:04 start to run EBPFProbeGNUTLS module
2022/06/05 15:32:04 invalid argument

bash-5.1# ecapture tls --pthread="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libpthread.so.0" --libssl="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libssl.so.1.0.2k"  --hex
2022/06/05 15:29:01 pid info :3255607
2022/06/05 15:29:01 start to run EBPFProbeOPENSSL module
2022/06/05 15:29:01 start to run EBPFProbeGNUTLS module
2022/06/05 15:29:01 invalid argument

[cfc4n]

may be it's a bug. can you send a new issue for it?

[vincentmli]

sure, will do, appreciate the help!

[vincentmli]

opened https://github.com/ehids/ecapture/issues/69

[vincentmli]

close this since question is answered by @cfc4n