search

cloudflare / quiche

🥧 Savoury implementation of the QUIC transport protocol and HTTP/3
https://docs.quic.tech/quiche/
BSD 2-Clause "Simplified" License
9.29k stars 697 forks source link

Cores on nginx-quiche while running curl command #1609

Open Karthikdasari0423 opened 1 year ago

Karthikdasari0423 commented 1 year ago

Hi ,

I tried to build nginx with quiche as explained below and tried to run curl with http3 but noticing below cores [https://github.com/cloudflare/quiche/tree/master/nginx]

nginx -V
root@ubuntu:/tmp/cores# /src/nginx-1.16.1/objs/nginx -V
nginx version: nginx/1.16.1 (quiche-ab2b44fa)
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --prefix=/src/nginx-1.16.1 --build=quiche-ab2b44fa --with-http_ssl_module --with-http_v2_module --with-http_v3_module --with-openssl=../quiche/quiche/deps/boringssl --with-quiche=../quiche

curl command and version

root@ubuntu:/tmp/cores# curl -V
curl 8.3.0-DEV (x86_64-pc-linux-gnu) libcurl/8.3.0-DEV OpenSSL/1.1.1s zlib/1.2.11 brotli/1.0.9 msh3/0.6.0.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe UnixSockets
root@ubuntu:/tmp/cores# curl -k -v --http3-only -# -o /tmp/index.html https://127.0.0.1:5443/index.html

below is the nginx bt

(gdb) bt
#0  0x000055d6712442c2 in ngx_quic_write_handler (wev=<optimized out>) at src/event/ngx_event_quic.c:411
#1  0x000055d67123456f in ngx_event_process_posted (cycle=cycle@entry=0x55d6737d0700, posted=0x55d6717a0280 <ngx_posted_events>)
    at src/event/ngx_event_posted.c:34
#2  0x000055d67123405c in ngx_process_events_and_timers (cycle=cycle@entry=0x55d6737d0700) at src/event/ngx_event.c:267
#3  0x000055d67123bf51 in ngx_worker_process_cycle (cycle=0x55d6737d0700, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:750
#4  0x000055d67123a431 in ngx_spawn_process (cycle=cycle@entry=0x55d6737d0700,
    proc=proc@entry=0x55d67123be67 <ngx_worker_process_cycle>, data=data@entry=0x4, name=name@entry=0x55d6715d8236 "worker process",
    respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:199
#5  0x000055d67123b537 in ngx_start_worker_processes (cycle=cycle@entry=0x55d6737d0700, n=8, type=type@entry=-3)
    at src/os/unix/ngx_process_cycle.c:359
#6  0x000055d67123c62d in ngx_master_process_cycle (cycle=cycle@entry=0x55d6737d0700) at src/os/unix/ngx_process_cycle.c:131
#7  0x000055d671214e0f in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:382
(gdb) quit

nginx error log file

2023/09/06 11:26:06 [alert] 240245#0: worker process 240272 exited on signal 11 (core dumped)
2023/09/06 11:26:08 [alert] 240245#0: worker process 240274 exited on signal 11 (core dumped)
2023/09/06 11:26:12 [alert] 240245#0: worker process 240276 exited on signal 11 (core dumped)
2023/09/06 11:26:26 [alert] 240245#0: worker process 240247 exited on signal 11 (core dumped)
2023/09/06 11:26:27 [alert] 240245#0: worker process 240303 exited on signal 11 (core dumped)
2023/09/06 11:26:29 [alert] 240245#0: worker process 240305 exited on signal 11 (core dumped)
2023/09/06 11:26:33 [alert] 240245#0: worker process 240307 exited on signal 11 (core dumped)
2023/09/06 11:31:05 [alert] 240245#0: worker process 240250 exited on signal 11 (core dumped)
2023/09/06 11:31:06 [alert] 240245#0: worker process 241691 exited on signal 11 (core dumped)
2023/09/06 11:31:08 [alert] 240245#0: worker process 241692 exited on signal 11 (core dumped)
2023/09/06 11:31:12 [alert] 240245#0: worker process 241693 exited on signal 11 (core dumped)
2023/09/06 11:38:52 [alert] 240245#0: worker process 241694 exited on signal 11 (core dumped)
2023/09/06 11:38:53 [alert] 240245#0: worker process 241734 exited on signal 11 (core dumped)
2023/09/06 11:38:55 [alert] 240245#0: worker process 241735 exited on signal 11 (core dumped)
2023/09/06 11:38:59 [alert] 240245#0: worker process 241736 exited on signal 11 (core dumped)
root@ubuntu:/tmp/cores#

am i missing anything here?

Ryenum commented 9 months ago

Hello, I encountered some problems when configuring the quic service of NGINX. My configuration is the same as that of the official website, but I still cannot use the quic protocol when accessing the server. The h2 protocol is still used. Here is my compilation information:

nginx version: nginx/1.16.1 (quiche-83d9168a) built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL) TLS SNI support enabled configure arguments: --prefix=/usr/local/nginx --build=quiche-83d9168a --with-http_v3_module --with-http_v2_module --with-quiche=/quiche --with-http_ssl_module --with-openssl=/quiche/quiche/deps/boringssl --add-module=/connect

Karthikdasari0423 commented 9 months ago

i build nginx with below args

root@ubuntu:~# /src/nginx-quiche/objs/nginx -V
nginx version: nginx/1.16.1 (quiche-cf2a087)
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --prefix=/src/nginx-quiche --build=quiche-cf2a087 --with-http_ssl_module --with-http_v2_module --with-http_v3_module --with-openssl=../quiche/deps/boringssl --with-quiche=../quiche
root@ubuntu:~#

Can you share the nginx conf file and command you are using to connect to nginx?

Ryenum commented 9 months ago

This is my configuration information：

server {

Enable QUIC and HTTP/3.

    listen 443 quic reuseport;
    server_name  test.cn;

    # Enable HTTP/2 (optional).
    listen 443 ssl http2;

    ssl_certificate      /usr/local/nginx/conf/cert/test.pem;
    ssl_certificate_key  /usr/local/nginx/conf/cert/test.key;

    # Enable all TLS versions (TLSv1.3 is required for QUIC).
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;

    # Add Alt-Svc header to negotiate HTTP/3.
    add_header alt-svc 'h3=":443"; ma=86400';

}

Can you help me see what the problem is?

Karthikdasari0423 commented 9 months ago

can you add below header also 

add_header X-protocol $server_protocol always;

and can you send netstat -alpn | grep nginx output

Ryenum commented 9 months ago

Here are the details：

root@learn:/quiche# netstat -alpn |grep nginx tcp 0 0 0.0.0.0:80 0.0.0.0: LISTEN 20157/nginx: master tcp 0 0 0.0.0.0:8080 0.0.0.0: LISTEN 20157/nginx: master udp 0 0 0.0.0.0:443 0.0.0.0:* 20157/nginx: master unix 3 [ ] 流 已连接 928126 20157/nginx: master
unix 3 [ ] 流 已连接 928127 20157/nginx: master

Karthikdasari0423 commented 9 months ago

can you try below config it works for me

              # to use the same port for quic and https
              listen 5443 quic reuseport;
              listen [::]:5443 quic reuseport;
              listen 5443 ssl;

              ssl_certificate     /etc/ssl/certs/ssl-cert-snakeoil.pem;
              ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
              ssl_protocols       TLSv1.3;
              ssl_ciphers         ALL:COMPLEMENTOFALL;
              ssl_session_cache   shared:SSL:10m;
              ssl_session_tickets on;
              ssl_session_timeout 5m;
              #to enable 0-rtt
              ssl_early_data      off;
              #to change to /var/www/html/
              root  /var/www/html/;

              location / {
                  # required for browsers to direct them into quic port
                  add_header Alt-Svc 'h3=":$server_port"; ma=86400';
                  #add_header Alt-Svc 'h3=":5443"; ma=86400';

                  # signal whether we are using QUIC+HTTP/3
                  add_header X-protocol $server_protocol always;

              }
Ryenum commented 9 months ago

After using this configuration, he compiled without problems, but the access times error "404".Is there something I'm missing?

Karthikdasari0423 commented 9 months ago

can you send me client output

Ryenum commented 9 months ago

Here are details:

"192.168.227.1 - - [04/Dec/2023:20:12:08 +0800] "GET / HTTP/1.1" 404 555 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" 192.168.227.1 - - [04/Dec/2023:20:14:18 +0800] "GET / HTTP/2.0" 404 555 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36""

Karthikdasari0423 commented 9 months ago

seems to me this is nginx error log file output

Ryenum commented 9 months ago

I don't quite understand what you mean. Here is the message I get back when I pass the "crul" command, hopefully I understood it correctly：

C:\Users\dell>curl -v https://ryetong.cn

C:\Users\dell>curl -v https://test.cn

Karthikdasari0423 commented 9 months ago

i think you didn't even tried with http3

try as curl --http3-only -v -k https://ryetong.cn:5443/

Karthikdasari0423 commented 9 months ago

and does your curl support quic? curl -V output ?

Ryenum commented 9 months ago

Sorry, my "curl" does not support http3：

curl: option --http3-only: the installed libcurl version doesn't support this curl: try 'curl --help' for more information

Karthikdasari0423 commented 9 months ago

try to install curl with h3 support

pplabs-fute commented 9 months ago

@Karthikdasari0423 Have you solved this coredump problem? How to solve it?

Karthikdasari0423 commented 9 months ago

@pplabs-fute yes,just install with https://github.com/bagder/quiche.git instaed of original one

pplabs-fute commented 9 months ago

@Karthikdasari0423 ok，thanks;

Hi-Im-Jony commented 5 months ago

@pplabs-fute yes,just install with https://github.com/bagder/quiche.git instaed of original one

Hey, I've had this issue too for a long time and it seems that using this fork to build nginx fixes the issue - so thanks haha. Do you have any idea why this fork works but the official repo doesn't? And how did you find out this fork works?

Karthikdasari0423 commented 5 months ago

@bagder Seems to me you have deleted https://github.com/bagder/quiche.git Could you pls add back if you archived or made it private? Many people are using your repo as it is stable in building nginx with quiche support

bagder commented 5 months ago

Sorry, my fork is gone.

Karthikdasari0423 commented 5 months ago

okay, got it. Thanks @bagder

mpiraux commented 4 months ago

I've been looking into this issue and was able to solve it by changing this piece of code: https://github.com/cloudflare/quiche/blob/1780aeceb686c212afdd2732b8a568cf5193f035/nginx/nginx-1.16.patch#L1050-L1054

The segfault comes from the quic_blocked_events queue which is uninitialised. But the fix is not to initialise it per se.

The real problem seems to me that consulting the write event of the listening connection is not appropriate to determine whether the connection socket is ready to write.

Changing the if condition to if (!c->write->ready) makes the code run and the QUIC connection go on.

@bagder could you kindly confirm this is the fix you undertook? Thanks :)

bagder commented 4 months ago

@bagder could you kindly confirm this is the fix you undertook? Thanks :)

I did not (on purpose) fix this. I suspect maybe my fork was just out of date.

Karthikdasari0423 commented 4 months ago

@mpiraux Does your below fork works for building nginx with quiche support? https://github.com/mpiraux/quiche and i have a backup of @bagder fork and as @bagder said it is out of date.

root@ubuntu:~/quiche# git log
commit cf2a08757c942d13f15a5a22aa7ea9ef50309cbe (HEAD -> master, origin/master, origin/HEAD)
Author: Junho Choi <junho@cloudflare.com>
Date:   Thu May 27 15:56:09 2021 -0700

    cubic: fix cwnd growth during congestion avoidance

    cwnd_inc is used for storing cwnd increments during congestion
    avoidance. When cwnd_inc >= MSS we increase cwnd by 1 MSS.
    Currently we clear cwnd_inc when cwnd is updated but this will
    lead to slightly slower growth because the residual part is gone.

commit 059b3d9c333ba61b2dc01f5e14ef95badca4fa03
Author: Junho Choi <1229714+junhochoi@users.noreply.github.com>
Date:   Fri May 28 01:50:36 2021 -0700

    don't try to send HANDSHAKE_DONE on the client

    `HANDSHAKE_DONE` can only be sent by a server, so don't try to send a packet from the client if the frame hasn't been sent yet.
Karthikdasari0423 commented 4 months ago

I've been looking into this issue and was able to solve it by changing this piece of code:

https://github.com/cloudflare/quiche/blob/1780aeceb686c212afdd2732b8a568cf5193f035/nginx/nginx-1.16.patch#L1050-L1054

The segfault comes from the quic_blocked_events queue which is uninitialised. But the fix is not to initialise it per se.

The real problem seems to me that consulting the write event of the listening connection is not appropriate to determine whether the connection socket is ready to write.

Changing the if condition to if (!c->write->ready) makes the code run and the QUIC connection go on.

@bagder could you kindly confirm this is the fix you undertook? Thanks :)

and this code is not even present in @bagder fork