Open dkonyshev opened 3 years ago
The server code does not work as it is. I get error regarding missing os
import, if I include that I get error regarding
os.makedirs(dir, exist_ok=True)
line in the server.
Sorry about that. Just fixed the code in the comment.
I see the same lines printed again and again. Could you post the output for
print(sock.getsockopt(zmq.SNDHWM))
before while True
in the server code.
print(sock.getsockopt(zmq.SNDHWM))
yields 1000.
Here is console output demonstrating the server used memory (6th number in the statm output) growing over time:
$ python3 zmq-send.py & while :; do cat /proc/$!/statm; sleep 5; done
[2] 1335088
2451 211 187 660 0 171 0
1000
27515 4258 2608 660 0 6417 0
27515 4588 2608 660 0 7620 0
27515 4984 2608 660 0 8809 0
27515 5380 2608 660 0 10006 0
27515 5776 2608 660 0 11192 0
I see a similar output when I reduce the sleep duration but after a point it remains constant. I guess that it just takes that long to grow to the true size. I am not sure why it takes so long for you for me it reaches the limit withing 1 sec
(venv)$ python3 server.py & while :; do cat /proc/$!/statm; sleep 0.02; done
[15] 21437
5326 442 338 660 0 2197 0
6153 1811 821 660 0 2977 0
7979 2669 1185 660 0 3446 0
1000
28461 2796 1298 660 0 23928 0
28461 2796 1298 660 0 23928 0
28461 2796 1298 660 0 23928 0
28461 2796 1298 660 0 23928 0
I left the test running over a night and the server used memory still keeps growing:
10005371 3028276 2592 660 0 9988941 0
10005371 3028672 2592 660 0 9990120 0
10005371 3029002 2592 660 0 9991316 0
10005371 3027685 2592 660 0 9992512 0
10005371 3028015 2592 660 0 9993708 0
10005371 3028411 2592 660 0 9994895 0
top
says the server takes 73,8% of system memory at the moment.
My system is Ubuntu 20.04:
Current libzmq version is 4.3.2 Current pyzmq version is 18.1.1
Not sure what's going on. I am using the following versions:
I don't see any memory increase with the versions you mentioned on WSL.
This is really confusing.
Besides Ubuntu 20.04, I observe the same behavior with memory allocated by the server indefinitely:
I tried to reproduce the given problem and caught the memory growth too. I probably think, that problem is in epoll() function of zmq thread.
If you allocate memory by using context = zmq.Context()
and don't destroy this context, valgrind will show a lot of leaks, one is interested:
==220861== 164,000 bytes in 10 blocks are still reachable in loss record 88 of 90
==220861== at 0x4C31DFB: malloc (vg_replace_malloc.c:309)
==220861== by 0x40A21A5: allocate_chunk (yqueue.hpp:189)
==220861== by 0x40A21A5: yqueue_t (yqueue.hpp:68)
==220861== by 0x40A21A5: zmq::pipe_t::hiccup() (ypipe.hpp:51)
==220861== by 0x40ADE97: zmq::session_base_t::reconnect() (session_base.cpp:519)
==220861== by 0x40ADFBF: zmq::session_base_t::engine_error(zmq::stream_engine_t::error_reason_t) (session_base.cpp:437)
==220861== by 0x40BA003: zmq::stream_engine_t::error(zmq::stream_engine_t::error_reason_t) (stream_engine.cpp:986)
==220861== by 0x40BC167: zmq::stream_engine_t::in_event() (stream_engine.cpp:321)
==220861== by 0x409114B: zmq::epoll_t::loop() [clone .part.11] (epoll.cpp:198)
==220861== by 0x40C250C: thread_routine (thread.cpp:182)
==220861== by 0x504E6DA: start_thread (pthread_create.c:463)
==220861== by 0x5EAD71E: clone (clone.S:95)
Do you see the same results if you do not run the zmq-rcv.py
code?
ipc
address, which I think also does not happen because the by the time it connects, the socket is also closed.I tried to reproduce the given problem and caught the memory growth too. I probably think, that problem is in epoll() function of zmq thread.
FWIW, this is the strace -p PID
output when a new connection is accepted.
epoll_wait(8, [{EPOLLIN, {u32=872418144, u64=140356108684128}}], 256, -1) = 1
accept4(10, NULL, NULL, SOCK_CLOEXEC) = 11
getpeername(11, {sa_family=AF_UNIX}, [128->2]) = 0
getsockname(11, {sa_family=AF_UNIX, sun_path="/tmp/my"}, [128->10]) = 0
getpeername(11, {sa_family=AF_UNIX}, [128->2]) = 0
getsockopt(11, SOL_SOCKET, SO_PEERCRED, {pid=95371, uid=1000, gid=1000}, [12]) = 0
fcntl(11, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(11, F_SETFL, O_RDWR|O_NONBLOCK) = 0
write(7, "\1\0\0\0\0\0\0\0", 8) = 8
epoll_wait(8, [{EPOLLIN, {u32=30834208, u64=30834208}}], 256, -1) = 1
poll([{fd=7, events=POLLIN}], 1, 0) = 1 ([{fd=7, revents=POLLIN}])
read(7, "\1\0\0\0\0\0\0\0", 8) = 8
epoll_ctl(8, EPOLL_CTL_ADD, 11, {0, {u32=872421776, u64=140356108687760}}) = 0
epoll_ctl(8, EPOLL_CTL_MOD, 11, {EPOLLIN, {u32=872421776, u64=140356108687760}}) = 0
epoll_ctl(8, EPOLL_CTL_MOD, 11, {EPOLLIN|EPOLLOUT, {u32=872421776, u64=140356108687760}}) = 0
recvfrom(11, "\377\0\0\0\0\0\0\0\1\177", 12, 0, NULL, NULL) = 10
recvfrom(11, "", 2, 0, NULL, NULL) = 0
epoll_ctl(8, EPOLL_CTL_DEL, 11, 0x7fa734001994) = 0
close(11) = 0
poll([{fd=7, events=POLLIN}], 1, 0) = 0 (Timeout)
epoll_wait(8,
Couldn't find a link quickly, but this sounds like another discussion from a year or two ago about, IIRC, zero-copy mechanism to speed up communications. Effectively, many network packets come into a buffer, pointers to insides of that are given as zmq protocol message contents, and the big buffer is only released when no references remain alive.
In that other thread, people saw large memory consumptions because (I think) did not quickly process and free the messages.
The older non-zerocopy can be enabled somehow, it is less stressful on memory at a CPU time cost of copying each message into its own buffer.
On Thu, Sep 16, 2021, 14:50 Giampaolo Rodola @.***> wrote:
I tried to reproduce the given problem and caught the memory growth too. I probably think, that problem is in epoll() function of zmq thread.
FWIW, this is the strace -p PID output when a new connection is accepted.
epoll_wait(8, [{EPOLLIN, {u32=872418144, u64=140356108684128}}], 256, -1) = 1 accept4(10, NULL, NULL, SOCK_CLOEXEC) = 11 getpeername(11, {sa_family=AF_UNIX}, [128->2]) = 0 getsockname(11, {sa_family=AF_UNIX, sun_path="/tmp/my"}, [128->10]) = 0 getpeername(11, {sa_family=AF_UNIX}, [128->2]) = 0 getsockopt(11, SOL_SOCKET, SO_PEERCRED, {pid=95371, uid=1000, gid=1000}, [12]) = 0 fcntl(11, F_GETFL) = 0x2 (flags O_RDWR) fcntl(11, F_SETFL, O_RDWR|O_NONBLOCK) = 0 write(7, "\1\0\0\0\0\0\0\0", 8) = 8 epoll_wait(8, [{EPOLLIN, {u32=30834208, u64=30834208}}], 256, -1) = 1 poll([{fd=7, events=POLLIN}], 1, 0) = 1 ([{fd=7, revents=POLLIN}]) read(7, "\1\0\0\0\0\0\0\0", 8) = 8 epoll_ctl(8, EPOLL_CTL_ADD, 11, {0, {u32=872421776, u64=140356108687760}}) = 0 epoll_ctl(8, EPOLL_CTL_MOD, 11, {EPOLLIN, {u32=872421776, u64=140356108687760}}) = 0 epoll_ctl(8, EPOLL_CTL_MOD, 11, {EPOLLIN|EPOLLOUT, {u32=872421776, u64=140356108687760}}) = 0 recvfrom(11, "\377\0\0\0\0\0\0\0\1\177", 12, 0, NULL, NULL) = 10 recvfrom(11, "", 2, 0, NULL, NULL) = 0 epoll_ctl(8, EPOLL_CTL_DEL, 11, 0x7fa734001994) = 0 close(11) = 0 poll([{fd=7, events=POLLIN}], 1, 0) = 0 (Timeout) epoll_wait(8,
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/zeromq/libzmq/issues/4189#issuecomment-920872293, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMPTFADKJTN56BWBVQ7CDDUCHRY7ANCNFSM4446U42A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Hello,
Could someone please explain what is wrong with my server code below? The problem is that the server keeps allocating memory every time a short-living client connects to its IPC socket but never seems to release that memory when the clients exit. In my real embedded system app the servers ends up devouring all the system memory and the system crashes.
I narrowed down the problem code to this server code:
And the client code:
Then, I run the server in one terminal as follows to keep track of used memory:
python3 zmq-send.py & while :; do cat /proc/$!/statm; sleep 5; done
And the clients are launching in another terminal as follows:
while :; do python3 zmq-rcv.py; done
It can be seen in the server memory printouts that its used memory grows very fast but never shrinks.
Any help will be very much appreciated.
Regards, Dmitry