nginx 1.9.9 not shutting down properly when on 1.10.33.2-beta

centminmod commented 8 years ago

Just updated on CentOS 6.7, nginx 1.9.9 to 1.10.33.2-beta using the clang and devtoolset-3 gcc 4.9.1 workaround outlined at https://github.com/pagespeed/ngx_pagespeed/issues/1079 on my live forums running pagespeed at https://community.centminmod.com/

After update, and restarting nginx gives me the following error

Starting nginx: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:81 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:82 failed (98: Address already in use)
nginx: [emerg] still could not bind()

and ends up with failed nginx start, process list shows nginx processes are still running so stop didn't happen.

If I turn off ngx_pagespeed via pagespeed off;, nginx starts properly.

Had no problems with ngx_pagespeed 1.10.33.1-beta previous beta

centminmod commented 8 years ago

more investigation seems it is related to the 2nd method outlined at https://github.com/pagespeed/ngx_pagespeed/issues/1079 with export CFLAGS workaround as even downgrading to 1.10.33.1-beta didn't help nginx properly stop and start

1.10.33.1-beta worked with the initial workaround passing devtoolset-3 toolchain via exported CC variable.

crowell commented 8 years ago

so it works when built with gcc, but not clang?

centminmod commented 8 years ago

yeah with GCC 4.9.1 not clang

i see even with nginx stop command that processes are consuming cpu time

top -c
top - 19:27:00 up 112 days, 13:26,  1 user,  load average: 6.18, 4.12, 2.56
Tasks: 186 total,   4 running, 182 sleeping,   0 stopped,   0 zombie
Cpu0  : 99.7%us,  0.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4045132k total,  3384436k used,   660696k free,    73224k buffers
Swap:  4194300k total,   287800k used,  3906500k free,  1401772k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                                     
 5634 nginx     10 -10  707m  39m  12m R 139.2  1.0   4:39.67 nginx: worker process                                                                                                                                                                       
 5633 nginx     10 -10  707m  39m  12m R 132.6  1.0   4:21.58 nginx: worker process                                                                                                                                                                       
 5635 nginx     10 -10  707m  39m  12m R 126.6  1.0   4:25.87 nginx: worker process

i am recompiling nginx with 1.10.33.2-beta with gcc 4.9.1 devtoolset now to confirm

centminmod commented 8 years ago

with GCC 4.9.1 nginx starts and forums load but all css is broken https://community.centminmod.com/ with 1.10.33.2-beta when pagespeed off; is set ?? with pagespeed on; css isn't broken

crowell commented 8 years ago

what is broken with the CSS? on a cursory look it appears fine to me.

centminmod commented 8 years ago

Looks like it's working now with pagespeed off and pagespeed on for css files.. must of been some caching within the forum app.

To summarize so far

nginx 1.9.9 with ngx_pagespeed 1.10.33.1-beta and 1.10.33.2-beta on CentOS 6.7

with clang and any gcc-toolchain path devtoolset-3 or CC exported gcc-toolchain path workaround with GCC 4.9.1 AND pagespeed on; results in nginx failing to properly stop and those Address already in use errors and nginx worker processes consuming large amounts of cpu resources. If pagespeed off; is set, nginx stops correctly and starts correctly. The confirmations at https://github.com/pagespeed/ngx_pagespeed/issues/1079 was with pagespeed off; on fresh installs hence I didn't see the problem until doing it live on my forums with pagespeed on;
with GCC compiled nginx via devtoolset-3, ngx_pagespeed 1.10 works fine and is currently in use and seems to be working on the forums with either pagespeed on; or pagespeed off;

the working GCC 4.9.1 devtoolset-3 compiled Nginx configuration

nginx -V
nginx version: nginx/1.9.9
built by gcc 4.9.1 20140922 (Red Hat 4.9.1-10) (GCC) 
built with LibreSSL 2.2.5
TLS SNI support enabled
configure arguments: --with-ld-opt='-lrt -ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-m64 -mtune=native -DTCP_FASTOPEN=23 -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-http_stub_status_module --with-http_secure_link_module --with-openssl-opt=enable-tlsext --add-module=../nginx-module-vts --with-libatomic --with-threads --with-stream --with-stream_ssl_module --with-http_gzip_static_module --add-module=../ngx_pagespeed-release-1.10.33.2-beta --with-http_sub_module --with-http_addition_module --with-http_image_filter_module --with-http_geoip_module --with-http_realip_module --add-module=../nginx-accesskey-2.0.3 --add-module=../nginx-http-concat-master --add-module=../ngx-fancyindex-ngx-fancyindex --add-module=../ngx_cache_purge-2.3 --add-module=../ngx_devel_kit-0.2.19 --add-module=../set-misc-nginx-module-0.29 --add-module=../echo-nginx-module-0.58 --add-module=../redis2-nginx-module-0.12 --add-module=../ngx_http_redis-0.3.7 --add-module=../lua-nginx-module-0.9.19 --add-module=../lua-upstream-nginx-module-0.04 --add-module=../lua-upstream-cache-nginx-module-0.1.1 --add-module=../nginx_upstream_check_module-0.3.0 --add-module=../openresty-memc-nginx-module-4f6f78f --add-module=../openresty-srcache-nginx-module-ffa9ab7 --add-module=../headers-more-nginx-module-0.28 --with-pcre=../pcre-8.38 --with-pcre-jit --with-http_ssl_module --with-http_v2_module --with-openssl=../libressl-2.2.5

Also confirmed this on my local test virtualbox CentOS 6.7 server for clang issues and devtoolset-3 GCC, same high nginx worker process cpu load, unable to stop nginx processes when pagespeed on; is set

arielesp commented 8 years ago

I would just like to inform you that I am also having several problems with this last 1.10.XX version , nginx 1.9.x.

After the upgrade, the server keeps crashing .

At the moment , I have to use version 1.9.32.11

jeffkaufman commented 8 years ago

@arielesp Does it print anything in the error log when it crashes?

arielesp commented 8 years ago

So, dont crash stopping... but dont answer and need restart nginx

I already tested in 20 servers...

There are some bug in version 1.10... or with compiler ... I dont know...

This problem occurs with big sites ...

server setup 32GB RAM SSD 120GB Intel Xeon E3 1245v2 CentOS 6.7 x64

I tested with nginx 1.9.9, only frontend / backend

And other with apache as backend

jeffkaufman commented 8 years ago

@arielesp It sounds to me like you're saying you're seeing 1.10 hang and be unresponsive, but it's not crashing, is that right? Is there anything printed to the error logs when it goes unresponsive?

arielesp commented 8 years ago

@jeffkaufman I didnt checked. ... but I will reinstall again to check ...

jeffkaufman commented 8 years ago

@centminmod Over on #1114 you indicated that you were still having these problems with 1.10, and so are still on 1.9?

(I had thought this was working for you now.)

centminmod commented 8 years ago

@jeffkaufman yes i have it working in latest beta of my Centmin Mod LEMP stack if i use GCC devtoolset but I need for it to work with Clang too as for my stack the difference in compile speed and memory usage is enough to justify using Clang compiler and that doesn't work with ngx_pagespeed 1.10.

For now, I am just temporarily disabling ngx_pagespeed from integration into the stable version of my Centmin Mod LEMP stack (which doesn't have my fix from my beta version) as it's breaking a lot of users nginx 1.9.11 upgrades (~3,000 new installs per month). Guess this is expected with nginx mainline 1.9 usage :)

crowell commented 8 years ago

@centminmod sorry for the delay on this.

I've tried with building clang 3.7, with using devtoolset and with gcc5 compiled from source.

Each time it seems like the __init_array isn't populated when using clang. I'm opening a bug with clang to track this (maybe it's still our fault, but I'm not convinced otherwise).

crowell commented 8 years ago

bug i opened on llvm's bugtracker https://llvm.org/bugs/show_bug.cgi?id=26727

centminmod commented 8 years ago

@crowell cheers much appreciated - fingers crossed :)

apache / incubator-pagespeed-ngx

nginx 1.9.9 not shutting down properly when on 1.10.33.2-beta #1082