coredump with specific URL and rewrite location in nginx.conf

[Georgisim]

Hi guys,

I've found a situation where I can crash vod-module with specific request:

george@george-desktop:~/work/git/merge/vod/nginx-1.17.10$ curl -v http://127.0.0.1:8088/hls/hls/some_video.mp4/master.m3u8/index.m3u8

nginx version is 1.17.10 and vod module is 1.27 tag

here is my minimal config where I can still reproduce it:

george@george-desktop:~/work/git/merge/vod/nginx-1.17.10/run$ cat conf/nginx.conf

user user user;
worker_processes 1;
pid ./nginx.pid;

error_log logs/error.log debug;
#error_log /dev/null;

worker_rlimit_nofile 204800;
worker_rlimit_core 8192M;
working_directory ./tmp;

events {
    worker_connections 24576;
    use epoll;
}

http {

    include mime.types;
    default_type application/octet-stream;

    #------------------------
    # VOD - hls and dash
    #------------------------

    vod_mode remote;
    vod_upstream_location /origin;

    #------------------------
    # Main server block
    #------------------------

    server {
        listen 127.0.0.1:8088 backlog=1024;

        vod_multi_uri_suffix /urlset;

        location /hls/ {
            vod hls;

            proxy_cache off;
        }

        # internal location for vod subrequests
        location /origin/ {
            internal;

            rewrite ^/origin/(hls|dash)(/.*)$ $2;           
        }

        location / {
            internal;
            proxy_pass http://127.0.0.1/8082;

            proxy_set_header Host $http_host;
        }
    }
}

This is the backtrace:

george@george-desktop:~/work/git/merge/vod/nginx-1.17.10/run$ gdb ./sbin/nginx tmp/core 
GNU gdb (Ubuntu 9.1-0ubuntu1) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./sbin/nginx...
[New LWP 2864594]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker process          '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000555b1570d3b0 in ?? ()
(gdb) bt 
#0  0x0000555b1570d3b0 in ?? ()
#1  0x0000555b154eeb4c in ngx_child_request_wev_handler (r=0x555b1570d400) at nginx-vod-module/ngx_child_http_request.c:235
#2  0x0000555b154388f7 in ngx_http_run_posted_requests (c=0x7fa4d1f481e0) at src/http/ngx_http_request.c:2389
#3  0x0000555b1543688e in ngx_http_process_request_headers (rev=0x7fa4d1d070d0) at src/http/ngx_http_request.c:1501
#4  0x0000555b15435bb1 in ngx_http_process_request_line (rev=0x7fa4d1d070d0) at src/http/ngx_http_request.c:1151
#5  0x0000555b15434316 in ngx_http_wait_request_handler (rev=0x7fa4d1d070d0) at src/http/ngx_http_request.c:500
#6  0x0000555b15410919 in ngx_epoll_process_events (cycle=0x555b156e2ff0, timer=60000, flags=1) at src/event/modules/ngx_epoll_module.c:901
#7  0x0000555b153fbf64 in ngx_process_events_and_timers (cycle=0x555b156e2ff0) at src/event/ngx_event.c:247
#8  0x0000555b1540d7e5 in ngx_worker_process_cycle (cycle=0x555b156e2ff0, data=0x0) at src/os/unix/ngx_process_cycle.c:750
#9  0x0000555b15409ac2 in ngx_spawn_process (cycle=0x555b156e2ff0, proc=0x555b1540d6f4 <ngx_worker_process_cycle>, data=0x0, name=0x555b1556ba93 "worker process", respawn=-3) at src/os/unix/ngx_process.c:199
#10 0x0000555b1540c413 in ngx_start_worker_processes (cycle=0x555b156e2ff0, n=1, type=-3) at src/os/unix/ngx_process_cycle.c:359
#11 0x0000555b1540b981 in ngx_master_process_cycle (cycle=0x555b156e2ff0) at src/os/unix/ngx_process_cycle.c:131
#12 0x0000555b153c279e in main (argc=3, argv=0x7ffc99716338) at src/core/nginx.c:382
(gdb) f 1 
#1  0x0000555b154eeb4c in ngx_child_request_wev_handler (r=0x555b1570d400) at nginx-vod-module/ngx_child_http_request.c:235
235         ctx->callback(ctx->callback_context, rc, b, content_length);
(gdb) p *ctx
$1 = {callback = 0x555b1570d3b0, callback_context = 0x555b156e12a0, response_buffer = 0x0, response_chain = 0x0, upstream_headers = {last = 0x0, part = {elts = 0x555b157141e0, nelts = 0, next = 0x0}, size = 0, 
    nalloc = 0, pool = 0x0}, sr = 0x0, error_code = -1, original_write_event_handler = 0x0, original_context = 0x555b1571ddf0, dont_send_header = 0, send_header_result = 0}
(gdb) l
230     }
231 
232     if (ctx->callback != NULL)
233     {
234         // notify the caller
235         ctx->callback(ctx->callback_context, rc, b, content_length);
236     }
237     else
238     {
239         if (r->header_sent || ctx->dont_send_header)

It seems to me that ctx->callback points to something that is not beginning of function. Also upstream is NULL

(gdb) p u $1 = (ngx_http_upstream_t *) 0x0 (gdb)

It there is not rewrite location ( location /origin/ ) in my config - it does not crash. I suspect this fix is causing the issue [(https://github.com/kaltura/nginx-vod-module/commit/3d0ab1cbe32a5d1d3ad99c735b16be19e78dbd02#diff-9abbbcaa76d4d09c33e94eb383ecc384]) because without it, we will exit this function here :

line 109 in old code:

if (u == NULL)
    {       {
        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,           ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
            "ngx_child_request_wev_handler: unexpected, upstream is null");             "ngx_child_request_wev_handler: unexpected, upstream is null");
        return;         return;
    }

I have a simple workaround, but I'm not sure it's the best one and fit original original author.

If you need more info, just contact me.

[erankor]

The location pointed by vod_upstream_location is expected to be a location with proxy_pass, rewrite is indeed not supported. If the goal is to remove the /hls prefix it can be achieved easily without rewrite - for your example, something like this -

location ~ ^/origin/(hls|dash)(/.*)$ {
    proxy_pass http://127.0.0.1/8082/$2;
    proxy_set_header Host $http_host;
}