nginx 1.9.5 crashes with brotli enabled, when being curl'ed

[ghost]

Hi,

not sure, if this an upstream google/brotli issue, but since I'm using cloudflare's bundle I'll put it here first, maybe you guys have an idea what's going on:

I packaged nginx 1.9.5 for debian, with ngx_brotli_module-master.

nginx.conf

brotli on;
brotli_comp_level 6;
brotli_min_length 1000;

nginx sits behind nghttpx, a tls offloader and http2 proxy, communicating over http/1.1/no-tls with the nginx backend. When opening a page with multiple png images (rand() generated) with chrome or firefox, everything's fine, using gzip though. Opening the same page with curl, both HTTP/1.1 and HTTP2, nginx crashes immediately.

curl:

curl 7.45.0 (x86_64-pc-linux-gnu) libcurl/7.45.0 GnuTLS/3.3.18 zlib/1.2.8 libidn/1.32 libssh2/1.5.0 nghttp2/1.3.4 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets

gdb ./nginx /tmp/core:

Reading symbols from ./nginx...done.

warning: exec file is newer than core file.
[New LWP 16376]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker pr'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f9e2c55d8c0 in ngx_http_brotli_header_filter (r=0x7f9e2cb60820)
    at /home/daniel/code/nginx/nginx-1.9.5/debian/modules/ngx_brotli_module-master/ngx_http_brotli_filter_module.c:171
171         if (!ngx_strstrn(ae->value.data, "br", 1)) {
(gdb) bt full
#0  0x00007f9e2c55d8c0 in ngx_http_brotli_header_filter (r=0x7f9e2cb60820)
    at /home/daniel/code/nginx/nginx-1.9.5/debian/modules/ngx_brotli_module-master/ngx_http_brotli_filter_module.c:171
        h = 0x7f9e2cb60d88
        ae = 0x0
        ctx = 0x7f9e2cb60820
        conf = 0x7f9e2cb7e338
#1  0x00007f9e2c51cf32 in ngx_http_headers_filter (r=0x7f9e2cb60820) at src/http/modules/ngx_http_headers_filter_module.c:208
        value = {len = 134,
          data = 0x7f9e2cb7aefb "pin-sha256=\"sggymnso4MC+EGK/CjpEYzpHqNYgFoT+3AxbPG/06LM=\"; pin-sha256=\"qWRoFIB2OK8l7M4/nAKetrygsdoqgA5MgQpGKeKgVi0=\"; max-age=15768000"}
        i = 1
        safe_status = 1
        h = 0x7f9e2cb7afb0
        conf = 0x7f9e2cb7e378
#2  0x00007f9e2c51e11d in ngx_http_not_modified_header_filter (r=0x7f9e2cb60820) at src/http/modules/ngx_http_not_modified_filter_module.c:61
No locals.
#3  0x00007f9e2c4e0b3b in ngx_http_send_header (r=0x7f9e2cb60820) at src/http/ngx_http_core_module.c:1947
No locals.
#4  0x00007f9e2c50e6e4 in ngx_http_upstream_send_response (r=0x7f9e2cb60820, u=0x7f9e2cb65ce8) at src/http/ngx_http_upstream.c:2653
        tcp_nodelay = 32670
        n = 140317332341344
        rc = 140317328430168
        p = 0xc28
        c = 0x7f9e2cc300a0
        clcf = 0x7f9e2cbca4a8
#5  0x00007f9e2c50d53e in ngx_http_upstream_process_header (r=0x7f9e2cb60820, u=0x7f9e2cb65ce8) at src/http/ngx_http_upstream.c:2165
        n = 3112
        rc = 0
        c = 0x7f9e2cbca4a8
#6  0x00007f9e2c50b7fd in ngx_http_upstream_handler (ev=0x7f9e2cc00260) at src/http/ngx_http_upstream.c:1095
        c = 0x7f9e2cbca3d0
        r = 0x7f9e2cb60820
        u = 0x7f9e2cb65ce8
#7  0x00007f9e2c4c6d4a in ngx_event_process_posted (cycle=0x7f9e2cb46a20, posted=0x7f9e2c8517f0 <ngx_posted_events>) at src/event/ngx_event_posted.c:33
        q = 0x7f9e2cc002b0
        ev = 0x7f9e2cc00260
#8  0x00007f9e2c4c410d in ngx_process_events_and_timers (cycle=0x7f9e2cb46a20) at src/event/ngx_event.c:259
        flags = 3
        timer = 14400000
        delta = 1
#9  0x00007f9e2c4d32ac in ngx_worker_process_cycle (cycle=0x7f9e2cb46a20, data=0x1) at src/os/unix/ngx_process_cycle.c:753
        worker = 1
#10 0x00007f9e2c4cf231 in ngx_spawn_process (cycle=0x7f9e2cb46a20, proc=0x7f9e2c4d3192 <ngx_worker_process_cycle>, data=0x1,
    name=0x7f9e2c590403 "worker process", respawn=-3) at src/os/unix/ngx_process.c:198
        on = 1
        pid = 0
        s = 1
#11 0x00007f9e2c4d1dea in ngx_start_worker_processes (cycle=0x7f9e2cb46a20, n=2, type=-3) at src/os/unix/ngx_process_cycle.c:358
        i = 1
        ch = {command = 1, pid = 16375, slot = 0, fd = 3}
#12 0x00007f9e2c4d1337 in ngx_master_process_cycle (cycle=0x7f9e2cb46a20) at src/os/unix/ngx_process_cycle.c:130
        title = 0x7f9e2cbbfc8c "master process /usr/sbin/nginx"
        p = 0x7f9e2cbbfcaa ""
        size = 31
        i = 1
        n = 0
        sigio = 0
        set = {__val = {0 <repeats 16 times>}}
        itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}
        live = 0
        delay = 0
        ls = 0x0
        ccf = 0x7f9e2cb47b20
#13 0x00007f9e2c4947a5 in main (argc=1, argv=0x7ffec3c42b38) at src/core/nginx.c:415
        b = 0x7f9e00000001
        log = 0x7f9e2c84c7e0 <ngx_log>
        i = 47
        cycle = 0x7f9e2cb46a20
        init_cycle = {conf_ctx = 0x0, pool = 0x7f9e2cb46530, log = 0x7f9e2c84c7e0 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0,
            disk_full_time = 0, handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0,
          free_connections = 0x0, free_connection_n = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts = 0x0, nelts = 0,
            size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {elts = 0x0, nelts = 0,
            size = 0, nalloc = 0, pool = 0x0}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0},
          shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0,
          connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21, data = 0x7f9e2c58c25e "/etc/nginx/nginx.conf"},
          conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, data = 0x7f9e2c58c25e "/etc/nginx/nginx.conf"}, prefix = {len = 17,
            data = 0x7f9e2c58c24c "/usr/share/nginx/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        cd = 0x7f9e2c46ded0
        ccf = 0x7f9e2cb47b20

[vkrasnov]

What is the Accept-Encoding field you supply with curl?

[ghost]

Hi,

full output of curl -v, using HTTP/1.1:

*   Trying 127.0.0.1...
* Connected to remote.host (127.0.0.1) port 443 (#0)
* found 180 certificates in /etc/ssl/certs/ca-certificates.crt
* found 724 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: remote.host (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        [...]
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /http2/ HTTP/1.1
> Host: remote.host
> User-Agent: curl/7.45.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 21 Oct 2015 13:53:38 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Vary: Accept-Encoding
< Vary: Accept-Encoding
< Public-Key-Pins: pin-sha256="sggymnso4MC+EGK/CjpEYzpHqNYgFoT+3AxbPG/06LM="; pin-sha256="qWRoFIB2OK8l7M4/nAKetrygsdoqgA5MgQpGKeKgVi0="; max-age=15768000
< Server: nghttpx nghttp2/1.3.4
< Via: 1.1 nghttpx
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000; preload
< Front-End-Https: on
< X-UA-Compatible: IE=Edge,chrome=1

[ghost]

Looking at the gdb output, #1, could this be an issue when using quotes in the http headers? (pin-sha256="")? Not sure why brotli would care for this, but not sure why this would be in the core either if it's not related?

[vkrasnov]

I pushed a possible fix, can you check and tell me if it worked?

[ghost]

confirmed, no crash with both h1 and h2, thanks!

[vkrasnov]

Thank you