search

tokers / zstd-nginx-module

Nginx modules for the Zstandard compression
BSD 2-Clause "Simplified" License
196 stars 23 forks source link

endless loop #25

Closed jason-x-xu closed 1 year ago

jason-x-xu commented 1 year ago

@tokers 你好， 有个问题请教下， 使用这个模块在做压测时偶尔会重现死循环， 请帮忙看看什么原因， 具体worker进程堆栈如下:

gdb /export/servers/nginx-1.10.3/sbin/nginx 180995 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /home/export/servers/nginx-1.10.3/sbin/nginx...done. Attaching to program: /home/export/servers/nginx-1.10.3/sbin/nginx, process 180995 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpcre.so.0 Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libfreebl3.so Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff9f9ec000 0x00000000004e8f4c in ZSTD_compressStream_generic () Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.149.el6_6.9.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-37.el6_6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 openssl-1.0.1e-30.el6.11.x86_64 pcre-7.8-6.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt

0 0x00000000004e8f4c in ZSTD_compressStream_generic ()

1 0x00000000004eb560 in ZSTD_compressStream ()

2 0x00000000004df0a1 in ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:541

3 ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426

4 0x000000000041892f in ngx_output_chain (ctx=0x21a6240, in=0x7fff9f9d74d0) at src/core/ngx_output_chain.c:74

5 0x000000000046bcaf in ngx_http_copy_filter (r=0x2237dd0, in=0x7fff9f9d74d0) at src/http/ngx_http_copy_filter_module.c:152

6 0x000000000046197d in ngx_http_range_body_filter (r=0x2237dd0, in=) at src/http/modules/ngx_http_range_filter_module.c:623

7 0x000000000043f941 in ngx_http_output_filter (r=, in=) at src/http/ngx_http_core_module.c:1970

8 0x00000000004470e0 in ngx_http_send_special (r=0x2237dd0, flags=2) at src/http/ngx_http_request.c:3336

9 0x0000000000456ab3 in ngx_http_upstream_finalize_request (r=0x2237dd0, u=0x21a53c0, rc=-1) at src/http/ngx_http_upstream.c:4229

10 0x0000000000457462 in ngx_http_upstream_process_request (r=0x2237dd0, u=0x21a53c0) at src/http/ngx_http_upstream.c:3830

11 0x0000000000458c01 in ngx_http_upstream_process_upstream (r=0x2237dd0, u=0x21a53c0) at src/http/ngx_http_upstream.c:3733

12 0x0000000000457165 in ngx_http_upstream_handler (ev=) at src/http/ngx_http_upstream.c:1117

13 0x000000000042eca2 in ngx_event_process_posted (cycle=, posted=0xc042f0) at src/event/ngx_event_posted.c:33

14 0x000000000042eb1f in ngx_process_events_and_timers (cycle=0x21a03d0) at src/event/ngx_event.c:259

15 0x0000000000434a9c in ngx_worker_process_cycle (cycle=0x21a03d0, data=) at src/os/unix/ngx_process_cycle.c:753

16 0x0000000000433387 in ngx_spawn_process (cycle=0x21a03d0, proc=0x434a11 , data=0x1, name=0x543d05 "worker process", respawn=1) at src/os/unix/ngx_process.c:198

17 0x00000000004352a6 in ngx_reap_children (cycle=0x21a03d0) at src/os/unix/ngx_process_cycle.c:621

18 ngx_master_process_cycle (cycle=0x21a03d0) at src/os/unix/ngx_process_cycle.c:174

19 0x00000000004159d1 in main (argc=, argv=) at src/core/nginx.c:367

(gdb) b ngx_http_zstd_filter_module.c:428 Breakpoint 1 at 0x4df28f: file ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c, line 428. (gdb) p ctx->last No symbol "ctx" in current context. (gdb) s Single stepping until exit from function ZSTD_compressStream_generic, which has no line number information. 0x00000000004eb560 in ZSTD_compressStream () (gdb) s Single stepping until exit from function ZSTD_compressStream, which has no line number information. ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:546 546 if (ZSTD_isError(rc)) { (gdb) p ctx->last $1 = 0 (gdb) p ctx->in->next Cannot access memory at address 0x8 (gdb) p ctx->in $2 = (ngx_chain_t ) 0x0 (gdb) p ctx->in_buf->last $3 = (u_char ) 0x0 (gdb) p ctx->in_buf $4 = (ngx_buf_t ) 0x21a61f0 (gdb) p ctx->in $5 = (ngx_chain_t ) 0x0 (gdb) p ctx->out $6 = (ngx_chain_t ) 0x0 (gdb) p ctx->out_buf $7 = (ngx_buf_t ) 0x21a62b8 (gdb) p ctx->buffer_in $8 = {src = 0x0, size = 0, pos = 0} (gdb) p ctx->buffer_out $9 = {dst = 0x2234510, size = 4096, pos = 0} (gdb) s 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 565 if (ctx->action == NGX_HTTP_ZSTD_FILTER_COMPRESS) { (gdb) 566 ctx->action = NGX_HTTP_ZSTD_FILTER_FLUSH; (gdb) 569 ctx->redo = 1; (gdb) 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) 505 pos_out = ctx->buffer_out.pos; (gdb) 507 switch (ctx->action) { (gdb) 519 rc = ZSTD_flushStream(ctx->cstream, &ctx->buffer_out); (gdb) p ctx->buffer_out $10 = {dst = 0x2234510, size = 4096, pos = 0} (gdb) p ctx->buffer_in $11 = {src = 0x0, size = 0, pos = 0} (gdb) p ctx->out_buf $12 = (ngx_buf_t ) 0x21a62b8 (gdb) p ctx->in_buf $13 = (ngx_buf_t ) 0x21a61f0 (gdb) p ctx->in $14 = (ngx_chain_t ) 0x0 (gdb) p ctx->out $15 = (ngx_chain_t ) 0x0 (gdb) s 546 if (ZSTD_isError(rc)) { (gdb) 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 571 } else if (ctx->last && ctx->action != NGX_HTTP_ZSTD_FILTER_END) { (gdb) 580 ctx->action = NGX_HTTP_ZSTD_FILTER_COMPRESS; / restore / (gdb) 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) 505 pos_out = ctx->buffer_out.pos; (gdb) 507 switch (ctx->action) { (gdb) 541 rc = ZSTD_compressStream(ctx->cstream, &ctx->buffer_out, (gdb) 546 if (ZSTD_isError(rc)) { (gdb) 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 565 if (ctx->action == NGX_HTTP_ZSTD_FILTER_COMPRESS) { (gdb) 566 ctx->action = NGX_HTTP_ZSTD_FILTER_FLUSH; (gdb) 569 ctx->redo = 1; (gdb) 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) 505 pos_out = ctx->buffer_out.pos; (gdb) 507 switch (ctx->action) { (gdb) 519 rc = ZSTD_flushStream(ctx->cstream, &ctx->buffer_out); (gdb) 546 if (ZSTD_isError(rc)) { (gdb) 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 571 } else if (ctx->last && ctx->action != NGX_HTTP_ZSTD_FILTER_END) { (gdb) 580 ctx->action = NGX_HTTP_ZSTD_FILTER_COMPRESS; / restore / (gdb) 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) set ctx->last=1 (gdb) s ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) 505 pos_out = ctx->buffer_out.pos; (gdb) 507 switch (ctx->action) { (gdb) 541 rc = ZSTD_compressStream(ctx->cstream, &ctx->buffer_out, (gdb) 546 if (ZSTD_isError(rc)) { (gdb) 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 565 if (ctx->action == NGX_HTTP_ZSTD_FILTER_COMPRESS) { (gdb) 566 ctx->action = NGX_HTTP_ZSTD_FILTER_FLUSH; (gdb) 569 ctx->redo = 1; (gdb) 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) 505 pos_out = ctx->buffer_out.pos; (gdb) 507 switch (ctx->action) { (gdb) 519 rc = ZSTD_flushStream(ctx->cstream, &ctx->buffer_out); (gdb) 546 if (ZSTD_isError(rc)) { (gdb) 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 571 } else if (ctx->last && ctx->action != NGX_HTTP_ZSTD_FILTER_END) { (gdb) 572 ctx->redo = 1; (gdb) 573 ctx->action = NGX_HTTP_ZSTD_FILTER_END; (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) 505 pos_out = ctx->buffer_out.pos; (gdb) 507 switch (ctx->action) { (gdb) 529 rc = ZSTD_endStream(ctx->cstream, &ctx->buffer_out); (gdb) set ctx->last=1

这里一直重复compress->flush->compress, 设置ctx->last可以终止循环，终止循环worker没有异常。

tokers commented 1 year ago

@jason-x-xu It has been so long since I wrote this module. I just saw the code, and I want to know that did you see the local variable rc after calling ZSTD_compressStream and ZSTD_flushStream? Also, if possible, could you re-compile your Nginx with --with-debug, so that the debug logs can be shown?

jason-x-xu commented 1 year ago

@tokers

这是另外一次的堆栈， 情况相同， 打印了rc：

(gdb) s Single stepping until exit from function ZSTD_compressStream_generic, which has no line number information. 0x00000000004eb15d in ZSTD_flushStream () (gdb) s Single stepping until exit from function ZSTD_flushStream, which has no line number information. ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:546 546 if (ZSTD_isError(rc)) { (gdb) p rc $1 = 0 (gdb) s 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 571 } else if (ctx->last && ctx->action != NGX_HTTP_ZSTD_FILTER_END) { (gdb) p ctx->last $2 = 0 (gdb) p ctx->action $3 = 1 (gdb) s 580 ctx->action = NGX_HTTP_ZSTD_FILTER_COMPRESS; / restore / (gdb) p ctx->in_buf->pos $4 = (u_char ) 0x0 (gdb) p ctx->out_buf->last $5 = (u_char ) 0x2234510 "\270uC \337\177" (gdb) p pos_in $6 = 0 (gdb) p pos_out $7 = 0 (gdb) s 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) p ctx->out_buf $8 = (ngx_buf_t ) 0x21a62b8 (gdb) s ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) p r->state $9 = 0 (gdb) p r->done $10 = 0 (gdb) p r->waited $11 = 0 (gdb) p r->err_status $12 = 0 (gdb) p r->header_size $13 = 402 (gdb) p r->connection $14 = (ngx_connection_t ) 0x7fdf1dcf5470 (gdb) p r->connection->timedout $15 = 0 (gdb) p r->connection->error $16 = 0 (gdb) p r->connection->idle $17 = 0 (gdb) p r->connection->close $18 = 0 (gdb) p r->connection->reusable $19 = 0 (gdb) p r->connection->number $20 = 45294 (gdb) p r->connection->sent $21 = 0 (gdb) p r->connection->fd $22 = 23 (gdb) s ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) s ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) p rc $23 = (gdb) $24 = (gdb) s ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) p pos_in $25 = (gdb) $26 = (gdb) s 505 pos_out = ctx->buffer_out.pos; (gdb) p pos_out $27 = (gdb) s 507 switch (ctx->action) { (gdb) p ctx->action $28 = 0 (gdb) s 541 rc = ZSTD_compressStream(ctx->cstream, &ctx->buffer_out, (gdb) p rc $29 = (gdb) s 546 if (ZSTD_isError(rc)) { (gdb) p rc $30 = 131072 (gdb) p rc $31 = 131072 (gdb) s 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) p rc $32 = 131072 (gdb) s 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) p rc $33 = 131072 (gdb) s 565 if (ctx->action == NGX_HTTP_ZSTD_FILTER_COMPRESS) { (gdb) p ctx->action $34 = 0 (gdb) s 566 ctx->action = NGX_HTTP_ZSTD_FILTER_FLUSH; (gdb) s 569 ctx->redo = 1; (gdb) s 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) s ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) s ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) p ctx->flush $35 = 1 (gdb) p ctx->last $36 = 0 (gdb) p ctx->redo $37 = 1 (gdb) p ctx->buffer_in.pos $38 = 0 (gdb) p ctx->buffer_in.size $39 = 0 (gdb) s ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) s ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) p ctx->buffer_out.pos $40 = 0 (gdb) p ctx->buffer_out.size $41 = 4096 (gdb) s ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) s ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) p ctx->buffer_in.pos $42 = 0 (gdb) s 505 pos_out = ctx->buffer_out.pos; (gdb) p ctx->buffer_out.pos $43 = 0 (gdb) p ctx->action $44 = 1 (gdb) s 507 switch (ctx->action) { (gdb) s 519 rc = ZSTD_flushStream(ctx->cstream, &ctx->buffer_out); (gdb) s 546 if (ZSTD_isError(rc)) { (gdb) p ctx->buffer_out $45 = {dst = 0x2234510, size = 4096, pos = 0} (gdb) p rc $46 = 0 (gdb) s 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) p ctx->buffer_in.pos $47 = 0 (gdb) p pos_in $48 = 0 (gdb) p ctx->in_buf->pos $49 = (u_char ) 0x0 (gdb) p ctx->buffer_out.pos $50 = 0 (gdb) p pos_out $51 = 0 (gdb) p rc $52 = 0 (gdb) s 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) p ctx->out_buf->last $53 = (u_char ) 0x2234510 "\270uC \337\177" (gdb) p ctx->buffer_out.pos $54 = 0 (gdb) p pos_out $55 = 0 (gdb) p ctx->action $56 = 1 (gdb) s 562 ctx->redo = 0; (gdb) s 564 if (rc > 0) { (gdb) p ctx->last $57 = 0 (gdb) p ctx->action $58 = 1 (gdb) s 571 } else if (ctx->last && ctx->action != NGX_HTTP_ZSTD_FILTER_END) { (gdb) s 580 ctx->action = NGX_HTTP_ZSTD_FILTER_COMPRESS; / restore / (gdb) p ctx->action $59 = 1 (gdb) s 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) p ctx->buffer_out $60 = {dst = 0x2234510, size = 4096, pos = 0} (gdb) s ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) s ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) s ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) s ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) s ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) 505 pos_out = ctx->buffer_out.pos; (gdb) 507 switch (ctx->action) { (gdb) 541 rc = ZSTD_compressStream(ctx->cstream, &ctx->buffer_out, (gdb) 546 if (ZSTD_isError(rc)) { (gdb) 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 565 if (ctx->action == NGX_HTTP_ZSTD_FILTER_COMPRESS) { (gdb) 566 ctx->action = NGX_HTTP_ZSTD_FILTER_FLUSH; (gdb) 569 ctx->redo = 1; (gdb) 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) s ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) s ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) s ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) s ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) 505 pos_out = ctx->buffer_out.pos; (gdb) 507 switch (ctx->action) { (gdb) 519 rc = ZSTD_flushStream(ctx->cstream, &ctx->buffer_out); (gdb) 546 if (ZSTD_isError(rc)) { (gdb) 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 571 } else if (ctx->last && ctx->action != NGX_HTTP_ZSTD_FILTER_END) { (gdb) 580 ctx->action = NGX_HTTP_ZSTD_FILTER_COMPRESS; / restore / (gdb) 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) s ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) s ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) s ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) s ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) s 505 pos_out = ctx->buffer_out.pos; (gdb) s 507 switch (ctx->action) { (gdb) s 541 rc = ZSTD_compressStream(ctx->cstream, &ctx->buffer_out, (gdb) s 546 if (ZSTD_isError(rc)) { (gdb) 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 565 if (ctx->action == NGX_HTTP_ZSTD_FILTER_COMPRESS) { (gdb) 566 ctx->action = NGX_HTTP_ZSTD_FILTER_FLUSH; (gdb) 569 ctx->redo = 1; (gdb) 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) 505 pos_out = ctx->buffer_out.pos; (gdb) 507 switch (ctx->action) { (gdb) 519 rc = ZSTD_flushStream(ctx->cstream, &ctx->buffer_out); (gdb) 546 if (ZSTD_isError(rc)) { (gdb) 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 571 } else if (ctx->last && ctx->action != NGX_HTTP_ZSTD_FILTER_END) { (gdb) 580 ctx->action = NGX_HTTP_ZSTD_FILTER_COMPRESS; / restore / (gdb) 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) ngx_http_zstd_filter_add_data (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:627 627 if (ctx->buffer_in.pos < ctx->buffer_in.size (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:416 416 rc = ngx_http_zstd_filter_get_buf(r, ctx); (gdb) ngx_http_zstd_filter_get_buf (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:677 677 if (ctx->buffer_out.pos < ctx->buffer_out.size) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:426 426 rc = ngx_http_zstd_filter_compress(r, ctx); (gdb) ngx_http_zstd_filter_compress (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:504 504 pos_in = ctx->buffer_in.pos; (gdb) 505 pos_out = ctx->buffer_out.pos; (gdb) 507 switch (ctx->action) { (gdb) 541 rc = ZSTD_compressStream(ctx->cstream, &ctx->buffer_out, (gdb) 546 if (ZSTD_isError(rc)) { (gdb) 560 ctx->in_buf->pos += ctx->buffer_in.pos - pos_in; (gdb) 561 ctx->out_buf->last += ctx->buffer_out.pos - pos_out; (gdb) 562 ctx->redo = 0; (gdb) 564 if (rc > 0) { (gdb) 565 if (ctx->action == NGX_HTTP_ZSTD_FILTER_COMPRESS) { (gdb) 566 ctx->action = NGX_HTTP_ZSTD_FILTER_FLUSH; (gdb) 569 ctx->redo = 1; (gdb) 583 if (ngx_buf_size(ctx->out_buf) == 0) { (gdb) ngx_http_zstd_body_filter (r=0x2237dd0, in=) at ../zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:406 406 rc = ngx_http_zstd_filter_add_data(r, ctx); (gdb) q A debugging session is active.

jason-x-xu commented 1 year ago

Caused by extra code to enhance statistic

jkarni commented 9 months ago

We’ve also run into busy loops with this module. Any idea what more precisely was the issue @jason-x-xu ?

tokers commented 8 months ago

@jkarni You can create another issue to reflect your problem. This one was closed.

VladRassokhin commented 2 months ago

@jason-x-xu @tokers Would you mind wrapping logs and stacktraces into three backticks ('code block'), so this issue won't be shown as 'referenced in #25' in all other issues?