search

wandenberg / nginx-push-stream-module

A pure stream http push technology for your Nginx setup. Comet made easy and really scalable.
Other
2.22k stars 295 forks source link

unable to allocate message if pushing many messages in a short time #226

Closed figocui closed 8 years ago

figocui commented 8 years ago

Hi Wanden,

Thank you for this great module. This module helps our team a lot. I have a question, and let me describe my test firstly.

Nginx version: 1.5.1, http-push-stream-module: 0.3.5 The configuration of Nginx is wrote by my team member, as bellow: ---> push_stream_message_ttl 300; push_stream_shared_memory_size 500m;

location /broadcast/sub { push_stream_subscriber long-polling; set $push_stream_channels_path $arg_channel; } location /broadcast/pub { push_stream_publisher admin; set $push_stream_channel_id $arg_channel; } <--- As I know, push_stream_shared_memory_size is used to configure the share memory size for this module, here is 500M. Also I think is push_stream_message_ttl is useless, because the message is not stored if there is no subscriber.

I use a script to push many messages(1000 messages, and each message body size is 1024*1024 bytes) in a loop, but after about 246 message, the client will get "500 Internal Server Error". Please be noted that the number(246) is different for every time, but it's about 240 ~ 250 for each testing.

Here is the output of script: ---> [root@localhost ~]# python test.py 0 : 200 - {"channel": "ecce8820-b289-11e5-a87e-000c290b16fc1", "published_messages": "502", "stored_messages": "0", "subscribers": "0"}

......

246 : 200 - {"channel": "ecce8820-b289-11e5-a87e-000c290b16fc1", "published_messages": "748", "stored_messages": "0", "subscribers": "0"}

Traceback (most recent call last): File "test.py", line 9, in resp = urllib2.urlopen("http://localhost/broadcast/pub?channel=ecce8820-b289-11e5-a87e-000c290b16fc1", message) File "/usr/local/lib/python2.7/urllib2.py", line 127, in urlopen return _opener.open(url, data, timeout) File "/usr/local/lib/python2.7/urllib2.py", line 410, in open response = meth(req, response) File "/usr/local/lib/python2.7/urllib2.py", line 523, in http_response 'http', request, response, code, msg, hdrs) File "/usr/local/lib/python2.7/urllib2.py", line 448, in error return self._call_chain(_args) File "/usr/local/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(_args) File "/usr/local/lib/python2.7/urllib2.py", line 531, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 500: Internal Server Error <---

Also I have checked error.log of nginx, I found that: ---> 2016/01/25 06:55:13 [crit] 10923#0: ngx_slab_alloc() failed: no memory 2016/01/25 06:55:13 [error] 10923#0: *9696 push stream module: unable to allocate message in shared memory, client: 127.0.0.1, server: localhost, request: "POST /broadcast/pub?channel=ecce8820-b289-11e5-a87e-000c290b16fc1 HTTP/1.0", host: "127.0.0.1"" <---

But after a short few seconds(for example, 3 seconds), I run the script again, after push 240~250 messages, the client will get "500 Internal Error" again.

So my question:

Because the message is not stored by configuration, why this module will fail to allocate message in shared memory after 246 messages? I think there is no need to keep previous 246 messages(they should not use memory), so 247th message should have enough memory to use.

How this module deal the memory of such messages(no need to be stored, and also no subscribers)?

Great thanks. Figo

wandenberg commented 8 years ago

Hi Figo,

Let me correct one of your assumptions first. There is no relation between store a message and have an user on the channel. The only thing taken into consideration to store a message or not is if the location is configured to do it so.

The current implementation of the module always put the message on the shared memory when you publish it, because this is needed to make it be accessible from another worker.

In the case you have configured to store it, the message is appended to the channel queue, if not it will be moved to trash queue as soon as all workers have checked it as "no more needed". And after some seconds on the trash queue, the memory is released. What I mean is, any message use the shared memory even if will be discarded immediately.

So the behavior you are facing is correct. The published messages will use your shared memory, and after some time it will be released, and you will be able to publish more messages.

Of course, this scenario where the channel don't have any subscribers and you have set to not store messages could be optimized to even process the published memory. But it is a corner case and can be done by your application. For instance, you can use the events channel to monitor if there is someone on the channel or not, and only publish the message if there is someone on it.

figocui commented 8 years ago

Hello Wanden,

Thank you so much for your detail explanation. From your explanation, I know that current implementation uses "some time" to free the messages of the trash queue. So I will optimize my application to solve my problem.

Many thanks, Figo