search

gojue / ecapture

Capturing SSL/TLS plaintext without a CA certificate using eBPF. Supported on Linux/Android kernels for amd64/arm64.
https://ecapture.cc
Apache License 2.0
10.7k stars 1.16k forks source link

在多种操作系统发行版上无法捕获mysqld服务端的keylog #587

Closed BoltzmannWXD closed 1 week ago

BoltzmannWXD commented 1 month ago

在多种操作系统发行版上无法捕获mysqld服务端的keylog。 目前尝试过的操作系统有CentOS8、OpenEuler22、Ubuntu22。 具体信息如下：

CentOS8

Operating System: CentOS Linux 8 CPE OS Name: cpe:/o:centos:centos:8 Kernel: Linux 4.18.0-553.5.1.el8.x86_64 Architecture: x86-64

OpenEuler22

Operating System: openEuler 22.03 (LTS-SP3) Kernel: Linux 5.10.0-182.0.0.95.oe2203sp3.x86_64 Architecture: x86-64

Ubuntu22

Operating System: Ubuntu 22.04.4 LTS
Kernel: Linux 5.15.0-117-generic Architecture: x86-64

问题复现 在服务端开启ecapture ./ecapture tls -m key 然后再通过其他服务器的命令进行连接 mysql -h 192.168.xx.xx -u admin -p 发现无法正常捕获keylog。 查看debug日志效果如下：

sudo cat /sys/kernel/debug/tracing/trace_pipe

  connection-2342    [000] ....1  1735.963771: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :2295

  connection-2342    [000] ....1  1735.963783: bpf_trace_printk: TLS version :772

  connection-2342    [000] ....1  1735.963783: bpf_trace_printk: client_random: 0 0 0

  connection-2342    [000] ....1  1735.963784: bpf_trace_printk: cipher_suite_st pointer: 2f8

  connection-2342    [000] ....1  1735.963787: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0

  connection-2342    [000] ....1  1735.963788: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14

  connection-2342    [000] ....1  1735.963820: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :2295

  connection-2342    [000] ....1  1735.963821: bpf_trace_printk: TLS version :772

  connection-2342    [000] ....1  1735.963821: bpf_trace_printk: client_random: 0 0 0

  connection-2342    [000] ....1  1735.963822: bpf_trace_printk: cipher_suite_st pointer: 2f8

  connection-2342    [000] ....1  1735.963823: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0

  connection-2342    [000] ....1  1735.963824: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14

  connection-2342    [000] ....1  1735.963831: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :2295

  connection-2342    [000] ....1  1735.963831: bpf_trace_printk: TLS version :772

  connection-2342    [000] ....1  1735.963832: bpf_trace_printk: client_random: 0 0 0

  connection-2342    [000] ....1  1735.963832: bpf_trace_printk: cipher_suite_st pointer: 2f8

  connection-2342    [000] ....1  1735.963833: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0

  connection-2342    [000] ....1  1735.963834: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14

  connection-2342    [000] ....1  1735.963840: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :2295

  connection-2342    [000] ....1  1735.963840: bpf_trace_printk: TLS version :772

  connection-2342    [000] ....1  1735.963841: bpf_trace_printk: client_random: 0 0 0

  connection-2342    [000] ....1  1735.963841: bpf_trace_printk: cipher_suite_st pointer: 2f8

  connection-2342    [000] ....1  1735.963842: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0

  connection-2342    [000] ....1  1735.963843: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14

  connection-2342    [000] ....1  1735.963874: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :2295

  connection-2342    [000] ....1  1735.963875: bpf_trace_printk: TLS version :772

  connection-2342    [000] ....1  1735.963875: bpf_trace_printk: client_random: 0 0 0

  connection-2342    [000] ....1  1735.963875: bpf_trace_printk: cipher_suite_st pointer: 2f8

  connection-2342    [000] ....1  1735.963876: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0

  connection-2342    [000] ....1  1735.963877: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14

实际发起的tls连接版本是tls1.2，而捕获的却是772（tls.13）

mysql wirshark 截图

正常情况

通过当前机器发起的mysql连接是可以正常获取keylog的，查看debug日志效果如下：

       mysql-4860  [000] d... 111590.766726: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :4860

       mysql-4860  [000] d... 111590.766745: bpf_trace_printk: TLS version :772

       mysql-4860  [000] d... 111590.766746: bpf_trace_printk: client_random: 42 51 38

       mysql-4860  [000] d... 111590.766747: bpf_trace_printk: cipher_suite_st pointer: 2baf6d8

       mysql-4860  [000] d... 111590.766747: bpf_trace_printk: cipher_suite_st value: 34137b70

       mysql-4860  [000] d... 111590.766748: bpf_trace_printk: cipher_id: 50336514

       mysql-4860  [000] d... 111590.766749: bpf_trace_printk: *****master_secret*****: 0 0 0

       mysql-4860  [000] d... 111590.766818: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :4860

       mysql-4860  [000] d... 111590.766819: bpf_trace_printk: TLS version :772

       mysql-4860  [000] d... 111590.766820: bpf_trace_printk: client_random: 42 51 38

       mysql-4860  [000] d... 111590.766821: bpf_trace_printk: cipher_suite_st pointer: 2baf6d8

       mysql-4860  [000] d... 111590.766821: bpf_trace_printk: cipher_suite_st value: 34137b70

       mysql-4860  [000] d... 111590.766821: bpf_trace_printk: cipher_id: 50336514

       mysql-4860  [000] d... 111590.766822: bpf_trace_printk: *****master_secret*****: 0 0 0

       mysql-4860  [000] d... 111590.766848: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :4860

       mysql-4860  [000] d... 111590.766848: bpf_trace_printk: TLS version :772

       mysql-4860  [000] d... 111590.766849: bpf_trace_printk: client_random: 42 51 38

       mysql-4860  [000] d... 111590.766849: bpf_trace_printk: cipher_suite_st pointer: 2bae998

       mysql-4860  [000] d... 111590.766849: bpf_trace_printk: cipher_suite_st value: 34137b70

       mysql-4860  [000] d... 111590.766850: bpf_trace_printk: cipher_id: 50336514

       mysql-4860  [000] d... 111590.766850: bpf_trace_printk: *****master_secret*****: 0 0 0

       mysql-4860  [000] d... 111590.766857: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :4860

       mysql-4860  [000] d... 111590.766857: bpf_trace_printk: TLS version :772

       mysql-4860  [000] d... 111590.766857: bpf_trace_printk: client_random: 42 51 38

       mysql-4860  [000] d... 111590.766858: bpf_trace_printk: cipher_suite_st pointer: 2bae998

       mysql-4860  [000] d... 111590.766858: bpf_trace_printk: cipher_suite_st value: 34137b70

       mysql-4860  [000] d... 111590.766858: bpf_trace_printk: cipher_id: 50336514

       mysql-4860  [000] d... 111590.766859: bpf_trace_printk: *****master_secret*****: 0 0 0

该情况下发起的是tls1.3捕获的也是tls1.3，完全正确。 后续又在其他机器（CentOS8）部署了mysql-client和ecapture，发现能够正常捕获mysql-client发起的请求连接的keylog。

编译调试模式

刚开始尝试的时候用的是仓库里发行的 ecapture-v0.8.4-linux-amd64.tar.gz，后来发现无法正常使用，故在Ubuntu22上用代码进行了编译，并打开了debug模式：

make debug=1

发现了该问题。

结论

由此可以发现作为mysql客户端是可以正常捕获keylog的，但是作为服务端无法正常捕获mysqld的keylog。

cfc4n commented 1 month ago

请给一些详细的技术分析，讲清楚为什么mysqld的keylog模式无法生效，谢谢。

Please provide a detailed technical analysis, explaining clearly why the keylog mode of mysqld cannot take effect, thank you.

BoltzmannWXD commented 1 month ago

具体来说不仅限于mysqld，而是作为openssl服务端的问题。在gdb跟踪调试了openssl的SSL_accept函数后发现，只有在SSL_do_handshake函数返回后SSL参数才有正确的赋值。随后将openssl_masterkey.h里的"uprobe/SSL_write_key"调整成"uretprobe/SSL_write_key"，验证成功。

BoltzmannWXD commented 1 month ago

以下是调试服务端时候的日志

  connection-38592 [003] d... 410734.666741: bpf_trace_printk: uprobe/SSL_write_key

  connection-38592 [003] d... 410734.666761: bpf_trace_printk: new key :96c0, new: 14040

  connection-38592 [003] d... 410734.666774: bpf_trace_printk: uretprobe/SSL_write_key

  connection-38592 [003] d... 410734.666774: bpf_trace_printk: openssl uretprobe/SSL_write masterKey PID :38521

  connection-38592 [003] d... 410734.666781: bpf_trace_printk: TLS version :772

  connection-38592 [003] d... 410734.666782: bpf_trace_printk: client_random: 0 0 0

  connection-38592 [003] d... 410734.666782: bpf_trace_printk: cipher_suite_st pointer: 1f0

  connection-38592 [003] d... 410734.666786: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0

  connection-38592 [003] d... 410734.666787: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14

  connection-38592 [003] d... 410734.666790: bpf_trace_printk: uprobe/SSL_write_key

  connection-38592 [003] d... 410734.666794: bpf_trace_printk: uretprobe/SSL_write_key

  connection-38592 [003] d... 410734.666794: bpf_trace_printk: openssl uretprobe/SSL_write masterKey PID :38521

  connection-38592 [003] d... 410734.666795: bpf_trace_printk: TLS version :772

  connection-38592 [003] d... 410734.666795: bpf_trace_printk: client_random: 0 0 0

  connection-38592 [003] d... 410734.666796: bpf_trace_printk: cipher_suite_st pointer: 1f0

  connection-38592 [003] d... 410734.666797: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0

  connection-38592 [003] d... 410734.666798: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14

  connection-38592 [003] d... 410734.666815: bpf_trace_printk: uprobe/SSL_write_key

  connection-38592 [003] d... 410734.666820: bpf_trace_printk: uprobe/SSL_write_key

  connection-38592 [003] d... 410734.666823: bpf_trace_printk: uretprobe/SSL_write_key

  connection-38592 [003] d... 410734.666824: bpf_trace_printk: openssl uretprobe/SSL_write masterKey PID :38521

  connection-38592 [003] d... 410734.666824: bpf_trace_printk: TLS version :772

  connection-38592 [003] d... 410734.666824: bpf_trace_printk: client_random: 0 0 0

  connection-38592 [003] d... 410734.666825: bpf_trace_printk: cipher_suite_st pointer: 1f0

  connection-38592 [003] d... 410734.666826: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0

  connection-38592 [003] d... 410734.666826: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14

  connection-38592 [003] d... 410734.666837: bpf_trace_printk: uprobe/SSL_write_key

  connection-38592 [003] d... 410734.666840: bpf_trace_printk: uretprobe/SSL_write_key

  connection-38592 [003] d... 410734.666840: bpf_trace_printk: openssl uretprobe/SSL_write masterKey PID :38521

  connection-38592 [003] d... 410734.666841: bpf_trace_printk: TLS version :772

  connection-38592 [003] d... 410734.666841: bpf_trace_printk: client_random: 0 0 0

  connection-38592 [003] d... 410734.666842: bpf_trace_printk: cipher_suite_st pointer: 1f0

  connection-38592 [003] d... 410734.666843: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0

  connection-38592 [003] d... 410734.666844: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14

  connection-38592 [003] d... 410734.672146: bpf_trace_printk: uretprobe/SSL_write_key

  connection-38592 [003] d... 410734.672164: bpf_trace_printk: openssl uretprobe/SSL_write masterKey PID :38521

  connection-38592 [003] d... 410734.672168: bpf_trace_printk: TLS version :771

  connection-38592 [003] d... 410734.672169: bpf_trace_printk: client_random: 66 b7 fc

  connection-38592 [003] d... 410734.672169: bpf_trace_printk: master_key: 69 89 0

可以发现最后，拿到了client_random。

以下是ecapture的日志：

2024-08-10T19:50:35-04:00 INF AppName="eCapture(旁观者)"
2024-08-10T19:50:35-04:00 INF HomePage=https://ecapture.cc
2024-08-10T19:50:35-04:00 INF Repository=https://github.com/gojue/ecapture
2024-08-10T19:50:35-04:00 INF Author="CFC4N <cfc4ncs@gmail.com>"
2024-08-10T19:50:35-04:00 INF Description="Capturing SSL/TLS plaintext without a CA certificate using eBPF. Supported on Linux/Android kernels for amd64/arm64."
2024-08-10T19:50:35-04:00 INF Version=linux_amd64:0.8.4-20240709-7e1ad52:5.15.0-117-generic
2024-08-10T19:50:35-04:00 WRN ========== module starting. ==========
2024-08-10T19:50:35-04:00 INF Kernel Info=4.18.0 Pid=42473
2024-08-10T19:50:35-04:00 INF listen=localhost:28256
2024-08-10T19:50:35-04:00 WRN Your environment is like a container. We won't be able to detect the BTF configuration.
If eCapture fails to run, try specifying the BTF mode. use `-b 2` to specify non-CORE mode.
2024-08-10T19:50:35-04:00 INF https server starting...You can update the configuration file via the HTTP interface.
2024-08-10T19:50:35-04:00 INF BTF bytecode mode: CORE. btfMode=0
2024-08-10T19:50:35-04:00 INF master key keylogger has been set. eBPFProgramType=KeyLog keylogger=ecapture_openssl_key.og
2024-08-10T19:50:35-04:00 INF module initialization. isReload=false moduleName=EBPFProbeOPENSSL
2024-08-10T19:50:35-04:00 INF Module.Run()
2024-08-10T19:50:35-04:00 INF OpenSSL/BoringSSL version found origin versionKey="OpenSSL 1.1.1k" versionKeyLower="openssl 1.1.1k"
2024-08-10T19:50:35-04:00 INF HOOK type:Openssl elf ElfType=2 binrayPath=/lib64/libssl.so.1.1 masterHookFuncs=["SSL_get_wbio","SSL_in_before","SSL_do_handshake"]
2024-08-10T19:50:35-04:00 INF setupManagers eBPFProgramType=KeyLog
2024-08-10T19:50:35-04:00 INF BPF bytecode file is matched. bpfFileName=user/bytecode/openssl_1_1_1j_kern_core_less52.o
2024-08-10T19:50:35-04:00 INF perfEventReader created mapSize(MB)=4
2024-08-10T19:50:35-04:00 INF module started successfully. isReload=false moduleName=EBPFProbeOPENSSL
2024-08-10T19:50:55-04:00 INF CLIENT_RANDOM save success CLientRandom=66b7fcdf2479d7fc638418f4ca04ee63a61d3d119763aa4fc91f28db4c722d50 TlsVersion=TLS1_2_VERSION bytes=176 eBPFProgramType=KeyLog
cfc4n commented 1 month ago

具体来说不仅限于mysqld，而是作为openssl服务端的问题。在gdb跟踪调试了openssl的SSL_accept函数后发现，只有在SSL_do_handshake函数返回后SSL参数才有正确的赋值。随后将openssl_masterkey.h里的"uprobe/SSL_write_key"调整成"uretprobe/SSL_write_key"，验证成功。

看上去，SSL_do_handshake函数不是准确的HOOK点，有时候TLS握手还没完成。应该是选择一个更合适的HOOK函数，对吗？

BoltzmannWXD commented 1 month ago

是的

cfc4n commented 2 weeks ago

hi, @BoltzmannWXD 还有后续吗？