ml-ipowow commented 11 years ago

Hi, I noticed recently that nginx shutdown procedure is slower with nginx-push-stream-module compiled. If no comet connection the shutdown procedure will take a sec or two. With one or more comet connections the shutdown takes 5+ secs. Is this expected? The problem is that this impacts the 'nginx restart' sequencing.

The issue is that an nginx restart command causes the new master nginx to start before the old nginx master has shut down. A side effect is that when the old nginx master eventually dies it deletes the pid file - which is where the new nginx master had already written its pid (new pid). Hence there is no more pidfile for the new nginx process!!

If you have Monit running to monitor pidfile / nginx process it will keep restarting nginx because pid file goes missing. Does not play nicely.

We can add a timer in the 'restart nginx' procedure that waits until the old nginx has died and deleted the pidfile before starting the new nginx but that's like putting a band-aid.

Could you double-check the shutdown sequence and see if/why it's hanging after the 'stop' command.

root 3401 0.0 0.2 366120 1332 ? Ss 04:22 0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/nginx.conf nginx 3403 0.5 7.9 413000 48452 ? S 04:22 0:00 nginx: worker process is shutting down nginx 3404 0.5 7.9 413000 48452 ? S 04:22 0:00 nginx: worker process is shutting down root 3439 0.0 0.2 366120 1328 ? Ss 04:22 0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/nginx.conf nginx 3441 2.5 7.9 413000 48448 ? S 04:22 0:00 nginx: worker process nginx 3442 2.5 7.9 413000 48448 ? S 04:22 0:00 nginx: worker process

This is on a Linux Centos 5.x.

Can you please confirm this behaviour on your platform?

My conf:

HTTP PUSH STREAM module

push_stream_shared_memory_size 256M;

Number of characters in channel name

push_stream_max_channel_id_length 128;

subscriber may create channels on demand or only authorized

push_stream_authorized_channels_only off;

ping frequency

push_stream_ping_message_interval 20s;

connection ttl to enable recycle

push_stream_subscriber_connection_ttl 4m;

max messages to store in memory per channel

push_stream_max_messages_stored_per_channel 1;

message ttl in queue before it's expired

push_stream_message_ttl 1439m;

(end) HTTP PUSH STREAM module

This is with iframe comet connection.

wandenberg commented 11 years ago

Hi,

I tried to reproduce the problem without success. Can you provide a complete nginx.conf and a step by step how to reproduce the problem? Are you using any other third party module? (send me the nginx -V result please) Which nginx version are you using? Which push stream version are you using?

dctrwatson commented 11 years ago

I can confirm this happens on Ubuntu as well.

Though only when shared memory is multi-GB used but yours is limited to only 256M.

sebcante commented 11 years ago

i can confirm this happens nginx version: nginx/1.2.9 built by gcc 4.6.3 20120306 (Red Hat 4.6.3-2) (GCC) push stream 1.3.5

wandenberg commented 11 years ago

Hi @sebcante, do you have a procedure to how reproduce the problem on a systematic way?

sebcante commented 10 years ago

hi wanderberg,

so for the late answer, @ml-ipowow tweaked our nginx init script , restarting nginx is now working as expected

wandenberg commented 10 years ago

hi,

thanks for the feedback. I've made some improvements for the new 0.4.0 version too, to speed up the reload operation. If you can, take a look on 0.4.x branch, the new tag will be release soon.