Unable to decrypt QUIC traffic

[Karthikdasari0423]

Hi @tiandrey

Thank you writing such a wonderful project.

root@ubuntu:/var/log/nginx# ls -l total 28 -rw-r----- 1 www-data adm 412 Jan 6 20:03 access.log -rw-r----- 1 www-data adm 1230 Dec 31 20:05 access.log.1 -rw-r----- 1 www-data adm 198 Dec 30 13:02 access.log.2.gz -rw-r----- 1 www-data adm 429 Dec 25 15:15 access.log.3.gz -rw-r----- 1 www-data adm 0 Dec 31 00:00 error.log -rw-r----- 1 www-data adm 752 Dec 30 13:02 error.log.1 -rw-r----- 1 www-data adm 343 Dec 25 10:07 error.log.2.gz -rw-r--r-- 1 root root 483 Jan 6 20:03 sslkeys.log

I am able to see the ssl key log file but only a small issue is even after loding the ssl key log file into wirshark, i am seeing this issue

am i missing something here or Could you please help me

Thank you,

[tiandrey]

Can you share pcap and log file with keys?

[tiandrey]

Also please note that I've never checked if my module's variables can be used to write log files in format that Wireshark expects (https://wiki.wireshark.org/TLS#using-the-pre-master-secret), so may be you'll have to convert log records first.

[Karthikdasari0423]

sure here is the keylog file and pcap New folder (3).zip

curl_with_quic_ssl_key.log Could you please check

[Karthikdasari0423]

Also please note that I've never checked if my module's variables can be used to write log files in format that Wireshark expects (https://wiki.wireshark.org/TLS#using-the-pre-master-secret), so may be you'll have to convert log records first.

if i need to decrypt my pcap,now i have to convert ssl key log which is got from your module to wireshark format right

[Karthikdasari0423]

any idea how to convert your module created nginx ssl key log file to wireshark expected form

[Karthikdasari0423]

would it be possible to add something like this to your module https://github.com/bcgit/bc-csharp/issues/343

[tiandrey]

Looks like current variables can be used to decrypt only non-rsa traffic using log format CLIENT_RANDOM $sslkeylog_cr $sslkeylog_mk: https://gitlab.com/wireshark/wireshark/-/blob/fadb4207699d01b1759bdbcd6f32eb7594a65cad/epan/dissectors/packet-tls-utils.c#L6268-6305 To decrypt RSA traffic you need a whole bunch of other variables - exporter secret, handshake traffic secrets, traffice secrets. I'll look into it.

[Karthikdasari0423]

Thank you for replying back @tiandrey sure,

It will be a great help if you are able to add this feature(decrypt rsa traffic feature) to your module. Hope this might help https://git.lekensteyn.nl/peter/wireshark-notes/tree/src/sslkeylog.c

[tiandrey]

Sorry, but it seems that since version 1.1.0 of OpenSSL direct access to those variables is impossible, and there are no functions like SSL_get_client_random to access SERVER_HANDSHAKE_TRAFFIC_SECRET, SERVER_TRAFFIC_SECRET_0, CLIENT_HANDSHAKE_TRAFFIC_SECRET or CLIENT_TRAFFIC_SECRET_0 (these are required to decrypt TLSv1.3 traffic, EXPORTER_SECRET did not make difference, although it's inaccessible too). What I can recommend is to log client random ($sslkeylog_cr) with any additional request info you need (like client IP, request URI, timestamp and so on), and use libsslkeylog.so (https://security.stackexchange.com/questions/216065/extracting-openssl-pre-master-secret-from-nginx) to dump all secrets from libssl. Use client random to join two logs (in sslkeylog it's the first hex field in all lines).

[tiandrey]

would it be possible to add something like this to your module https://github.com/bcgit/bc-csharp/issues/343

No, because The Bouncy Castle Cryptography library is an implementation of cryptographic algorithms and protocols, and they can access anything they want in internal code because it's their implementation, while I'm limited to use of API provided by libssl. I don't want to patch openssl; may be it's possible to create wrapper library like libsslkeylog.so to implement required functionality - unfortunately I don't have enough free time to do this.

[Karthikdasari0423]

Sorry, but it seems that since version 1.1.0 of OpenSSL direct access to those variables is impossible, and there are no functions like SSL_get_client_random to access SERVER_HANDSHAKE_TRAFFIC_SECRET, SERVER_TRAFFIC_SECRET_0, CLIENT_HANDSHAKE_TRAFFIC_SECRET or CLIENT_TRAFFIC_SECRET_0 (these are required to decrypt TLSv1.3 traffic, EXPORTER_SECRET did not make difference, although it's inaccessible too). What I can recommend is to log client random ($sslkeylog_cr) with any additional request info you need (like client IP, request URI, timestamp and so on), and use libsslkeylog.so (https://security.stackexchange.com/questions/216065/extracting-openssl-pre-master-secret-from-nginx) to dump all secrets from libssl. Use client random to join two logs (in sslkeylog it's the first hex field in all lines).

i tried https://security.stackexchange.com/questions/216065/extracting-openssl-pre-master-secret-from-nginx this but it didnt worked cause the problem is my nginx is running with boringssl not openssl root@ubuntu:~# nginx -V nginx version: nginx/1.23.4 (nginx-quic) built by gcc 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1) built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL) TLS SNI support enabled

[Karthikdasari0423]

would it be possible to add something like this to your module bcgit/bc-csharp#343

No, because The Bouncy Castle Cryptography library is an implementation of cryptographic algorithms and protocols, and they can access anything they want in internal code because it's their implementation, while I'm limited to use of API provided by libssl. I don't want to patch openssl; may be it's possible to create wrapper library like libsslkeylog.so to implement required functionality - unfortunately I don't have enough free time to do this.

yeah,okay but it was great effort in this module too anyway once again Thank you for your help

[marcindulak]

As mentioned in the above comment https://github.com/tiandrey/nginx-sslkeylog/issues/2#issuecomment-1376786335, attempts to generate SSLKEYLOGFILE with the libsslkeylog.so ld_preload method for nginx as described at https://security.stackexchange.com/questions/216065/extracting-openssl-pre-master-secret-from-nginx with https://git.lekensteyn.nl/peter/wireshark-notes appear to fail (create empty SSLKEYLOGFILE), but https://github.com/drivenet/sslkeylog appears to still work.

Nginx closed the feature request to implement SSLKEYLOGFILE https://trac.nginx.org/nginx/ticket/2498 with the comment

nginx does not provide a way to capture SSL keys, consider using those from the client

On the other hand, SSLKEYLOGFILE feature is present in apache https://github.com/apache/httpd/pull/74. The comment says https://github.com/apache/httpd/blob/2.4.x/CHANGES#L738-L740

Support logging private key material for use with wireshark via log file given by SSLKEYLOGFILE environment variable. Requires OpenSSL 1.1.1

Could you comment on the differences in the implementation?

[Karthikdasari0423]

I tried to build this but building itself failed

https://github.com/drivenet/sslkeylog

cc  -O3 sslkeylog.c -Wall -Wpedantic -Wextra -shared -o libsslkeylog.so -fPIC -ldl
sslkeylog.c:18:10: fatal error: openssl/ssl.h: No such file or directory
   18 | #include <openssl/ssl.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:17: libsslkeylog.so] Error 1
root@ubuntu:~/sslkeylog/src#

This ticket was also raised by me and nginx said they wont support to capture SSL keys https://trac.nginx.org/nginx/ticket/2498

I didnt got you,what do you mean by differences?

[marcindulak]

In order to complile libssl is needed sudo apt install git make gcc libssl-dev.

The question about implementation differrences is for @tiandrey: apache mod_ssl uses OpenSSL 1.1.1, and generates values for SERVER_HANDSHAKE_TRAFFIC_SECRET, SERVER_TRAFFIC_SECRET_0, etc mentioned in https://github.com/tiandrey/nginx-sslkeylog/issues/2#issuecomment-1376133483

[Karthikdasari0423]

okay,Thank you Still i am not able to see any file being created

below is nginx conf

env LD_PRELOAD=/root/sslkeylog/src/libsslkeylog.so;
env SSLKEYLOGISSERVER=1;
env SSLKEYLOGFILE=/tmp/sslkeylog/nginx;

Below nginx.service file

root@ubuntu:/tmp/sslkeylog/nginx# cat /lib/systemd/system/nginx.service
[Unit]
Description=nginx - high performance web server
Documentation=https://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
ExecReload=/bin/sh -c "/bin/kill -s HUP $(/bin/cat /var/run/nginx.pid)"
ExecStop=/bin/sh -c "/bin/kill -s TERM $(/bin/cat /var/run/nginx.pid)"
Environment=LD_PRELOAD=/root/sslkeylog/src/libsslkeylog.so

[Install]
WantedBy=multi-user.target

Below is curl command i am trying

root@ubuntu:/tmp/sslkeylog/nginx# pwd
/tmp/sslkeylog/nginx
root@ubuntu:/tmp/sslkeylog/nginx# curl -v -k --http3-only -o /tmp/BPS.pdf -# https://127.0.0.1:8443/BPS.pdf

any idea what am i missing

[tiandrey]

On the other hand, SSLKEYLOGFILE feature is present in apache apache/httpd#74. The comment says https://github.com/apache/httpd/blob/2.4.x/CHANGES#L738-L740

Support logging private key material for use with wireshark via log file given by SSLKEYLOGFILE environment variable. Requires OpenSSL 1.1.1

Could you comment on the differences in the implementation?

@marcindulak, thanks for the link, I'll look into details of apache's implementation. May be libssl not just dropped access to private variables, but also added new functions to access them - their documentation is kinda absent, kinda shitty, a few or none code examples, so working code is much help.

[tiandrey]

Well, it seems that my module alone can't do the trick - I guess I'll have to patch ngx_ssl_module to add keylogging callback before actual handshaking happens, otherwise there will be no key data to log :( I intend to try to add this functionality, but it will obviously be a pain to keep up with nginx upstream development - new version, new patch.

[marcindulak]

My nginx systemd uses the override.conf as shown in https://security.stackexchange.com/questions/216065/extracting-openssl-pre-master-secret-from-nginx, but other than this it's similar to your. To make sure: your nginx is started after a sudo systemctl daemon-reload https://unix.stackexchange.com/questions/364782/what-does-systemctl-daemon-reload-do? The https://github.com/drivenet/sslkeylog generates a SSLKEYLOGFILE-timestamp file instead of the requestes SSLKEYLOGFILE filename, and has a different format (does not contain SERVER_HANDSHAKE_TRAFFIC_SECRET, SERVER_TRAFFIC_SECRET_0, but instead a log line) but still appears to decrypt TLSv1.3.

I'm still thinking about the closed issue https://trac.nginx.org/nginx/ticket/2498. Apache implements SSLKEYLOGFILE, and newer servers like https://github.com/envoyproxy/envoy/issues/10377 also do it, so maybe it's possible to at least ask nginx for clarification for the reasons why SSLKEYLOGFILE is not implemented.

[Karthikdasari0423]

That would be a great idea to ask them why they didnt provide when apache(compitetor) provided

[tiandrey]

I'm working on it, now I'm getting keylog callbacks and can parse those lines and store them in connection context (unfortunately, main portion of code is inside nginx, not inside module).

[tiandrey]

Well, it looks like it works. I'll add some finishing touches and push soon.

[Karthikdasari0423]

Okay, Thank you so much @tiandrey Please let me know once it is done And hope it works on latest nginx version also

[tiandrey]

@Karthikdasari0423 , please check out the latest commit.

[Karthikdasari0423]

@tiandrey does this support 1.25.x versions ?

[tiandrey]

The patch must apply fine, maybe with different offset. Just try it.

[Karthikdasari0423]

@tiandrey will try 1.24.0.patch on 1.25.3 version

[Karthikdasari0423]

@tiandrey able to build with nginx 1.25.3 and able to generate keylog file also but unable to decrypt pcap with keys generated

below are the keys file and pcap file test_nginx.zip sslkeys.log

am i missing anything here

[tiandrey]

Please describe your setup, ways to reproduce, etc.

[tiandrey]

I was able to decode my test capture with http2 requests encoded with tls1.2 and tls1.3 just fine. I need your nginx config and info on how you make requests in order to reproduce and see myself.

[Karthikdasari0423]

@tiandrey i was trying to decode http3 traffic and this traffic uses only tls1.3 below is my nginx conf and below are the steps to repro issue on your setup

steps

cd /root/
sudo apt install git gcc make g++  perl  libunwind-dev  golang  mercurial  libperl-dev  libpcre3-dev zlib1g-dev  libxslt1-dev  libgd-ocaml-dev  libgeoip-dev

cd /opt/
wget https://github.com/Kitware/CMake/releases/download/v3.27.4/cmake-3.27.4-linux-x86_64.sh
 chmod +x cmake-3.27.4-linux-x86_64.sh
 bash cmake-3.27.4-linux-x86_64.sh --skip-license --include-subdir --prefix=/opt
 ln -sf /opt/cmake-3.27.4-linux-x86_64/bin/* /usr/bin/
 cmake --version

 cd /root/
 git clone https://github.com/udhos/update-golang
 cd /root/update-golang
 ./update-golang.sh
 source /etc/profile.d/golang_path.sh

 mv /usr/local/go/bin/go /usr/bin/.
echo export GOROOT=/usr/local/go >> /root/.bashrc

sudo apt install curl gnupg2 ca-certificates lsb-release ubuntu-keyring
curl https://nginx.org/keys/nginx_signing.key | gpg --dearmor | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null
gpg --dry-run --quiet --no-keyring --import --import-options import-show /usr/share/keyrings/nginx-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/mainline/ubuntu `lsb_release -cs` nginx"  | sudo tee /etc/apt/sources.list.d/nginx.list

sudo apt install nginx
cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf_old
service nginx restart

mkdir /src/
cd /src/ 
git clone https://boringssl.googlesource.com/boringssl
mkdir /src/boringssl/build/
cd /src/boringssl/build/
cmake ..
make

cd  /src/
hg clone http://hg.nginx.org/njs -b 0.8.2

cd /root/
git clone https://github.com/tiandrey/nginx-sslkeylog.git

cd /src/
hg clone https://hg.nginx.org/nginx-quic -b release-1.25.3
cd /src/nginx-quic
patch -Np1 -i /root/nginx-sslkeylog/nginx-patches/1.24.0.patch
auto/configure `nginx -V 2>&1 | sed "s/ \-\-/ \\\ \n\t--/g" | grep -v -e 'http-geoip2' | grep "\-\-" | grep -ve opt= -e param= -e build=` --build=nginx-quic --with-debug  --with-http_v3_module --with-cc-opt="-I/src/boringssl/include" --with-ld-opt="-L/src/boringssl/build/ssl -L/src/boringssl/build/crypto" --add-module=/root/nginx-sslkeylog/
make
make install

now replace contents of /etc/nginx/nginx.conf with below file 
[nginx_conf.txt](https://github.com/tiandrey/nginx-sslkeylog/files/13242326/nginx_conf.txt)

service nginx restart

nginx -V

nginx conf file is below

https://github.com/tiandrey/nginx-sslkeylog/files/13242326/nginx_conf.txt

use below curl command

curl -v -k --http3-only -# -o /tmp/test.pdf https://localhost:8443/test.pdf

[tiandrey]

Please also show output of nginx -V and lsb_release -a.

And what version of curl are you using? Mine does not support QUIC, and I don't see any quick ways to install a cli QUIC client.

[tiandrey]

I've built curl version 8.4.0 according to this instruction: https://curl.se/docs/http3.html I've built ngint-1.25.3 with BoringSSL and with minimal subset of modules

./objs/nginx -V
nginx version: nginx/1.25.3
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) 
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --http-client-body-temp-path=/run/body_temp --http-proxy-temp-path=/run/proxy_temp --http-uwsgi-temp-path=/run/uwsgi_temp --http-scgi-temp-path=/run/scgi_temp --http-fastcgi-temp-path=/run/fastcgi_temp --with-threads --with-debug --with-http_ssl_module --with-poll_module --with-select_module --with-compat --with-http_v2_module --with-http_v3_module --with-cc-opt=-I/home/atikhonov/git/boringssl/include --with-ld-opt='-L/home/atikhonov/git/boringssl/build/ssl -L/home/atikhonov/git/boringssl/build/crypto' --add-module=/home/atikhonov/git/tiandrey/nginx-sslkeylog

I use the following server config: nginx-test.conf.txt

... and sorry, looks like it works for me, I'm unable to reproduce the problem - I see decrypted HTTP3 frames in Wireshark.

[tiandrey]

Try the following:

1. download fresh version of patch (https://github.com/tiandrey/nginx-sslkeylog/blob/master/nginx-patches/1.25.3.patch)
2. modify the following three lines (https://github.com/tiandrey/nginx-sslkeylog/blob/master/nginx-patches/1.25.3.patch#L40-L42) like this:

   +    /* uncomment to add more debugging info */
   +    ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL KEYLOG for connection %xl: \"%s\"", (unsigned long) c, line);
   +    /**/

   Apply patch, recompile nginx, add error_log /var/log/nginx/debug.log debug; to your config, and repeat the test. Then attach that debug log alongside packet dump and keylog file (and don't forget to show your nginx -V).

[Karthikdasari0423]

seems to me some issue

Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
|index 8468101d1..7e30df120 100644
|--- a/src/event/ngx_event_openssl.c
|+++ b/src/event/ngx_event_openssl.c
--------------------------
patching file src/event/ngx_event_openssl.c
Using Plan A...
Hunk #1 succeeded at 27.
Hunk #2 succeeded at 429.
patch: **** malformed patch at line 79: diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h

root@ubuntu:/src/nginx-quic#

[Karthikdasari0423]

and below is my nginx -V

root@ubuntu:/tmp# nginx -V
nginx version: nginx/1.25.3 (nginx-quic)
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-debug --with-http_v3_module --build=nginx-quic --with-debug --with-http_v3_module --with-cc-opt=-I/src/boringssl/include --with-ld-opt='-L/src/boringssl/build/ssl -L/src/boringssl/build/crypto' --add-module=/root/nginx-sslkeylog/
root@ubuntu:/tmp#

and i believe if traffic is going via localhost then we dont need any keys to decrypt traffic

my setup is like below

Ubuntu-A(client) ------------- Middle Boxes -------------- Ubuntu-B(server)
 (curl is running)                                                                (server is running)

[tiandrey]

Just apply this patch (to clean sources, not to already patched): nginx-with-debug-1.25.3.txt

and i believe if traffic is going via localhost then we dont need any keys to decrypt traffic

Wrong, if you are decrypting traffic dump - it does not matter whether it is from localhost or not, because encryption keys are never transmitted over network. However, traffic may be re-encrypted alongside the network (mitm), so it's crucial that you dump traffic on the same host where you log ssl keys.

[Karthikdasari0423]

applied nginx patch on a new clean sources and still not able to decrpypt below are pcaps and logs test_nginx_keys.zip test_nginx_keys.txt debug.log

Hope i am loading keys correctly, Wireshark --->Edit --->Preferences ---> Protocols ---->TLS ----> (Pre)-Master-Secret Log Filename