Lax / traffic-accounting-nginx-module

Monitor the incoming and outgoing traffic metrics in realtime for NGINX
https://github.com/Lax/traffic-accounting-nginx-module
BSD 2-Clause "Simplified" License
280 stars 77 forks source link

NGINX 1.16.1 - dynamic module keeps crashing ( segfault ) #42

Closed radixxko closed 5 years ago

radixxko commented 5 years ago

Hello, I've compiled the dynamic module with same flags as installed nginx binary but after loading the accounting module nginx keeps crashing.

Any hints? Is it possible to publish prebuild binaries for the NGINX version 1.16.1 ?

systemctl status nginx.service

kernel: Code: 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 08 00 00 00 0f 0b <48> 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 
kernel: nginx[572]: segfault at 0 ip 00007f725667a1b2 sp 00007fff7df3ba68 error 4 in ngx_http_accounting_module.so[7f725667a000+2000]
kernel: Code: 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 08 00 00 00 0f 0b <48> 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 
kernel: nginx[574]: segfault at 0 ip 00007f725667a1b2 sp 00007fff7df3ba68 error 4 in ngx_http_accounting_module.so[7f725667a000+2000]
kernel: Code: 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 08 00 00 00 0f 0b <48> 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 
kernel: nginx[576]: segfault at 0 ip 00007f725667a1b2 sp 00007fff7df3ba68 error 4 in ngx_http_accounting_module.so[7f725667a000+2000]
kernel: Code: 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 08 00 00 00 0f 0b <48> 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 
kernel: nginx[578]: segfault at 0 ip 00007f725667a1b2 sp 00007fff7df3ba68 error 4 in ngx_http_accounting_module.so[7f725667a000+2000]
kernel: Code: 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 08 00 00 00 0f 0b <48> 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 
kernel: nginx[580]: segfault at 0 ip 00007f725667a1b2 sp 00007fff7df3ba68 error 4 in ngx_http_accounting_module.so[7f725667a000+2000]
kernel: Code: 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 08 00 00 00 0f 0b <48> 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 
kernel: nginx[582]: segfault at 0 ip 00007f725667a1b2 sp 00007fff7df3ba68 error 4 in ngx_http_accounting_module.so[7f725667a000+2000]
kernel: Code: 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 08 00 00 00 0f 0b <48> 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 
kernel: nginx[584]: segfault at 0 ip 00007f725667a1b2 sp 00007fff7df3ba68 error 4 in ngx_http_accounting_module.so[7f725667a000+2000]
kernel: Code: 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 08 00 00 00 0f 0b <48> 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 
kernel: nginx[586]: segfault at 0 ip 00007f725667a1b2 sp 00007fff7df3ba68 error 4 in ngx_http_accounting_module.so[7f725667a000+2000]
kernel: Code: 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 08 00 00 00 0f 0b <48> 8b 04 25 00 00 00 00 0f 0b 48 8b 04 25 20 00 00 00 0f 0b 48

/var/log/nginx/error.log

2019/11/19 23:05:42 [alert] 31669#31669: worker process 32054 exited on signal 11 (core dumped)
2019/11/19 23:05:42 [alert] 31669#31669: worker process 32056 exited on signal 11 (core dumped)
2019/11/19 23:05:42 [alert] 31669#31669: worker process 32058 exited on signal 11 (core dumped)
2019/11/19 23:05:42 [alert] 31669#31669: worker process 32060 exited on signal 11 (core dumped)
2019/11/19 23:05:42 [alert] 31669#31669: worker process 32062 exited on signal 11 (core dumped)
2019/11/19 23:05:42 [alert] 31669#31669: worker process 32064 exited on signal 11 (core dumped)
2019/11/19 23:05:43 [alert] 31669#31669: worker process 32066 exited on signal 11 (core dumped)
2019/11/19 23:05:43 [alert] 31669#31669: worker process 32068 exited on signal 11 (core dumped)
2019/11/19 23:05:43 [alert] 31669#31669: worker process 32070 exited on signal 11 (core dumped)
2019/11/19 23:05:43 [alert] 31669#31669: worker process 32072 exited on signal 11 (core dumped)

accounting.log

2019/11/19 22:49:38 [notice] 10043#10043: pid:10043|start http traffic accounting
2019/11/19 22:49:38 [notice] 10045#10045: pid:10045|start http traffic accounting
2019/11/19 22:49:38 [notice] 10047#10047: pid:10047|start http traffic accounting
2019/11/19 22:49:38 [notice] 10049#10049: pid:10049|start http traffic accounting
2019/11/19 22:49:38 [notice] 10051#10051: pid:10051|start http traffic accounting
2019/11/19 22:49:38 [notice] 10053#10053: pid:10053|start http traffic accounting
2019/11/19 22:49:38 [notice] 10055#10055: pid:10055|start http traffic accounting
2019/11/19 22:49:38 [notice] 10057#10057: pid:10057|start http traffic accounting
2019/11/19 22:49:38 [notice] 10059#10059: pid:10059|start http traffic accounting
2019/11/19 22:49:38 [notice] 10061#10061: pid:10061|start http traffic accounting
2019/11/19 22:49:38 [notice] 10063#10063: pid:10063|start http traffic accounting
2019/11/19 22:49:38 [notice] 10065#10065: pid:10065|start http traffic accounting
wualex commented 5 years ago

i have the same issue, even on a fresh Ubuntu 18.04 system Did you find a workaround for this @radixxko?

@Lax any idea what could be the problem?

Lax commented 5 years ago

@wualex @radixxko can you reproduce this issue based on the Dockerfile in the fix-dynamic-module-crashing-ubuntu branch? https://github.com/Lax/traffic-accounting-nginx-module/tree/fix-dynamic-module-crashing-ubuntu

Maybe you need to update build flags and config files.

wualex commented 5 years ago

@Lax i did the following based on that Dockerfile

docker build -t nginxtest .

But i get an error while it builds

checking for getaddrinfo() ... found
configuring additional dynamic modules
adding module in traffic-accounting-nginx-module
./configure: error: no traffic-accounting-nginx-module/config was found
The command '/bin/sh -c ./configure --prefix=${PREFIX}     --with-stream     
--add-dynamic-module=traffic-accounting-nginx-module     
--add-dynamic-module=echo-nginx-module-master     
--http-log-path=/dev/stdout     --error-log-path=/dev/stderr    
 && make -s && make -s install' returned a non-zero code: 1

i don't have any experience with Docker so maybe i'm doing something wrong? Also it seems that the Dockerfile uses nginx 1.16.0 - while we are trying to run latest stable which is 1.16.1

thanks.

radixxko commented 5 years ago

@wualex I have changed it to 1.16.1 and I am running test currently, will get you posted

radixxko commented 5 years ago

It keeps failing for me "modules/ngx_http_accounting_module.so" failed ( undefined symbol: ngx_stream_get_variable_ind...

I will give it another try in a few hours

radixxko commented 5 years ago

@wualex Please try the attached dynamic module - it works for my NGINX 1.16.1 installation ngx_http_accounting_module.so.zip

I have built it with the following config inside docker container then pulled out the ngx_http_accounting_module.so

./configure --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-bTTM30/nginx-1.16.1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' \
    --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' \
    --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf \
    --http-log-path=/var/log/nginx/access.log \
    --error-log-path=/var/log/nginx/error.log \
    --lock-path=/var/lock/nginx.lock \
    --pid-path=/run/nginx.pid \
    --modules-path=/usr/lib/nginx/modules \
    --http-client-body-temp-path=/var/lib/nginx/body \
    --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
    --http-proxy-temp-path=/var/lib/nginx/proxy \
    --http-scgi-temp-path=/var/lib/nginx/scgi \
    --http-uwsgi-temp-path=/var/lib/nginx/uwsgi \ 
    --with-compat \
    --with-debug \
    --with-pcre-jit \
    --with-http_ssl_module \
    --with-http_stub_status_module \
    --with-http_realip_module \
    --with-http_auth_request_module \
    --with-http_v2_module \
    --with-http_dav_module \
    --with-http_slice_module \
    --with-threads \
    --with-http_addition_module \
    --with-http_geoip_module=dynamic \
    --with-http_gunzip_module \
    --with-http_gzip_static_module \
    --with-http_image_filter_module=dynamic \
    --with-http_sub_module \
    --with-http_xslt_module=dynamic \
    --with-mail=dynamic \
    --with-mail_ssl_module \
    --add-dynamic-module=traffic-accounting-nginx-module-master \
    --add-dynamic-module=echo-nginx-module-master \
    && make -s && make -s install
wualex commented 5 years ago

@radixxko i can confirm that it's working for me too, great job! thank you.