wandenberg
commented
12 years ago Hi Eugene,
Could you send me your configuration file? How much memory you set to shared memory?
I'm still investigating the way Nginx reuse memory, but it's not so easy.
I owe an estimate of how much memory each request costs in shared and no shared memory, sorry. I was very busy at work.
If you can repeat your tests using the last commit on github, please. There are some important changes since 0.3.1 version.
On Mon, Dec 12, 2011 at 6:28 AM, Eugene < reply@reply.github.com
wrote:
Hi all. We are using nginx-push-stream-module.
And we get next errors:
2011/12/11 18:52:23 [crit] 24244#0: ngx_slab_alloc() failed: no memory 2011/12/11 18:52:23 [error] 24244#0: 73289555 push stream module: unable to allocate worker subscriber queue marker in shared memory, client: 213.85.187.19, server: 172.17.0.4, request: "GET /xxx/v1/sub/106q2chk9s1ld1fd3m1hx9vuib" 2011/12/11 18:52:23 [crit] 24244#0: ngx_slab_alloc() failed: no memory 2011/12/11 18:52:23 [error] 24244#0: 73289563 push stream module: unable to allocate worker subscriber queue marker in shared memory, client: 213.85.187.19, server: 172.17.0.4, request: "GET
/xxx/v1/sub/34lgiys1st5ip0t0qkw81gcr?rnd=0.02243080991320312"
In view of the memory leaks in the push-module nginx in production, I spent a load testing comet. The test is written in Java. It creates a thread pool of 500 threads. Each of them is connected to the comet-channel. each time to the unique. The test worked through the night.
Now the results.
~ ENVIRONMENT: Platform: x86-64. uname-a: Linux server1 2.6.18-194.32.1.el5 # 1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU / Linux 4 processor with two cores in each = 8 nuclei. Version of nginx: 1.1.7. Version push_module: 0.3.1.
~ MEMORY CONSUMPTION No-load each nginx worker consumes ~ 42 248 KB c 500 compounds attached worker nginx-a consumption: 45 080 KB Hence, at 500 connections took 45 080 - 42 248 = 3560 KB Therefore, one compound is spent, empirically, 3560/500 ~ 7 kb. That is, on 50000 compounds have only ~ 350 mb.
Questions: Exactly how much memory is consumed on one channel? Exactly how much memory is consumed on a single connection? Exactly how much system memory is necessary for 50,000 connections?
The test worked through the night. Timeout on nginx stood at 1 minute. That is, flows constantly reconnect to random channels. By morning no leaks were found. all workers consume memory> 50 MB.
Hence comes the idea that the problem could be in production at the level of TCP / IP. We are writing middleware for IPTV. Clients use SetTopBoxes to connect. Maybe the operating system at STBs disconnects from nginx don't correct?
Question: What Linux kernel configuration recommened for use push_module? We are all time-outs are reduced to a minimum. TIMEWAITS connections should not be much.
I want to get tcpdump from our clients and publish its here.
I hope the above information will help to eliminate memory leaks.
P.S. cat /proc/net/sockstat sockets: used 829 TCP: inuse 658 orphan 0 tw 0 alloc 662 mem 504 UDP: inuse 12 mem 0 RAW: inuse 0 FRAG: inuse 0 memory 0
Reply to this email directly or view it on GitHub: https://github.com/wandenberg/nginx-push-stream-module/issues/9
JohnBat26
Hi all. We are using nginx-push-stream-module.
And we get next errors:
2011/12/11 18:52:23 [crit] 24244#0: ngx_slab_alloc() failed: no memory 2011/12/11 18:52:23 [error] 24244#0: *73289555 push stream module: unable to allocate worker subscriber queue marker in shared memory, client: 213.85.187.19, server: 172.17.0.4, request: "GET /xxx/v1/sub/106q2chk9s1ld1fd3m1hx9vuib" 2011/12/11 18:52:23 [crit] 24244#0: ngx_slab_alloc() failed: no memory
2011/12/11 18:52:23 [error] 24244#0: *73289563 push stream module: unable to allocate worker subscriber queue marker in shared memory, client: 213.85.187.19, server: 172.17.0.4, request: "GET /xxx/v1/sub/34lgiys1st5ip0t0qkw81gcr?rnd=0.02243080991320312"
In view of the memory leaks in the push-module nginx in production, I spent a load testing comet. The test is written in Java. It creates a thread pool of 500 threads. Each of them is connected to the comet-channel. each time to the unique. The test worked through the night.
Now the results.
~ ENVIRONMENT: Platform: x86-64. uname-a: Linux server1 2.6.18-194.32.1.el5 # 1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU / Linux 4 processor with two cores in each = 8 nuclei. Version of nginx: 1.1.7. Version push_module: 0.3.1.
~ MEMORY CONSUMPTION No-load each nginx worker consumes ~ 42 248 KB c 500 compounds attached worker nginx-a consumption: 45 080 KB Hence, at 500 connections took 45 080 - 42 248 = 3560 KB Therefore, one compound is spent, empirically, 3560/500 ~ 7 kb. That is, on 50000 compounds have only ~ 350 mb.
Questions: Exactly how much memory is consumed on one channel? Exactly how much memory is consumed on a single connection? Exactly how much system memory is necessary for 50,000 connections?
The test worked through the night. Timeout on nginx stood at 1 minute. That is, flows constantly reconnect to random channels. By morning no leaks were found. all workers consume memory> 50 MB.
Hence comes the idea that the problem could be in production at the level of TCP / IP. We are writing middleware for IPTV. Clients use SetTopBoxes to connect. Maybe the operating system at STBs disconnects from nginx don't correct?
Question: What Linux kernel configuration recommened for use push_module? We are all time-outs are reduced to a minimum. TIMEWAITS connections should not be much.
I want to get tcpdump from our clients and publish its here.
I hope the above information will help to eliminate memory leaks.
P.S. cat /proc/net/sockstat sockets: used 829 TCP: inuse 658 orphan 0 tw 0 alloc 662 mem 504 UDP: inuse 12 mem 0 RAW: inuse 0 FRAG: inuse 0 memory 0