search

OpenSIPS / opensips

OpenSIPS is a GPL implementation of a multi-functionality SIP Server that targets to deliver a high-level technical solution (performance, security and quality) to be used in professional SIP server platforms.
https://opensips.org
Other
1.26k stars 576 forks source link

[BUG] PKG memory issue #3434

Closed musicianx2 closed 1 month ago

musicianx2 commented 2 months ago

OpenSIPS version you are running

version: opensips 3.3.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: f2a083a01
main.c compiled on 07:24:34 Sep 26 2022 with gcc 9

Describe the bug I am using opensips as a sbc with topology_hiding module. Opensips processes always creating new fragments and using more private memory size. Each processes fragments count and real_used_size always increasing. Opensips has an average of 400 active calls at the moment. Opensips starting with -FE -m 4096 -M 64

Relevant System Logs

opensips-cli -x mi get_statistics pkmem:

"pkmem:1-total_size": 67108864, "pkmem:1-used_size": 9059640, "pkmem:1-real_used_size": 9184568, "pkmem:1-max_used_size": 9185144, "pkmem:1-free_size": 57924296, "pkmem:1-fragments": 1627, "pkmem:3-total_size": 67108864, "pkmem:3-used_size": 9062296, "pkmem:3-real_used_size": 9184592, "pkmem:3-max_used_size": 9184592, "pkmem:3-free_size": 57924272, "pkmem:3-fragments": 1580, "pkmem:6-total_size": 67108864, "pkmem:6-used_size": 10751392, "pkmem:6-real_used_size": 13273904, "pkmem:6-max_used_size": 13465808, "pkmem:6-free_size": 53834960, "pkmem:6-fragments": 40657, "pkmem:7-total_size": 67108864, "pkmem:7-used_size": 9341040, "pkmem:7-real_used_size": 9861048, "pkmem:7-max_used_size": 9986408, "pkmem:7-free_size": 57247816, "pkmem:7-fragments": 7927, ...

"pkmem:8-total_size": 67108864,
"pkmem:8-used_size": 14323928,
"pkmem:8-real_used_size": 21897584,
"pkmem:8-max_used_size": 22030000,
"pkmem:8-free_size": 45211280,
"pkmem:8-fragments": 123866,

(pid:14 id:8) opensips-cli -x mi mem_pkg_dump 14 -3 

{"log":"/usr/sbin/opensips[14]: Memory status (pkg):\r\n","stream":"stdout","time":"2024-07-09T08:33:07.400202202Z"}

{"log":"/usr/sbin/opensips[14]: fm_status (0x7f6ca5205010):\r\n","stream":"stdout","time":"2024-07-09T08:33:07.400272289Z"}

{"log":"/usr/sbin/opensips[14]:  heap size= 67108864\r\n","stream":"stdout","time":"2024-07-09T08:33:07.400308046Z"}

{"log":"/usr/sbin/opensips[14]:  used= 14319456, used+overhead=21886672, free=52789408\r\n","stream":"stdout","time":"2024-07-09T08:33:07.400340619Z"}

{"log":"/usr/sbin/opensips[14]:  max used (+overhead)= 22022336\r\n","stream":"stdout","time":"2024-07-09T08:33:07.40051667Z"}

{"log":"/usr/sbin/opensips[14]: dumping free list:\r\n","stream":"stdout","time":"2024-07-09T08:33:07.400581401Z"}

{"log":"/usr/sbin/opensips[14]: hash =   1 fragments no.:     2, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.400609905Z"}
{"log":"\u0009\u0009 bucket size:         8 -         8 (first         8)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.400623245Z"}

{"log":"/usr/sbin/opensips[14]: hash =   2 fragments no.:    53, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.400691692Z"}
{"log":"\u0009\u0009 bucket size:        16 -        16 (first        16)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.40072887Z"}

{"log":"/usr/sbin/opensips[14]: hash =   3 fragments no.: 10652, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.402865163Z"}
{"log":"\u0009\u0009 bucket size:        24 -        24 (first        24)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.402912046Z"}

{"log":"/usr/sbin/opensips[14]: hash =   6 fragments no.:     3, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.402957195Z"}
{"log":"\u0009\u0009 bucket size:        48 -        48 (first        48)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.402975715Z"}

{"log":"/usr/sbin/opensips[14]: hash =   7 fragments no.:    23, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403023503Z"}
{"log":"\u0009\u0009 bucket size:        56 -        56 (first        56)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403054643Z"}

{"log":"/usr/sbin/opensips[14]: hash =   8 fragments no.:     3, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403140967Z"}
{"log":"\u0009\u0009 bucket size:        64 -        64 (first        64)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403185111Z"}

{"log":"/usr/sbin/opensips[14]: hash =   9 fragments no.:    18, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403224692Z"}
{"log":"\u0009\u0009 bucket size:        72 -        72 (first        72)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403271888Z"}

{"log":"/usr/sbin/opensips[14]: hash =  12 fragments no.:     1, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403312358Z"}
{"log":"\u0009\u0009 bucket size:        96 -        96 (first        96)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403326325Z"}

{"log":"/usr/sbin/opensips[14]: hash =  16 fragments no.:     3, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403429775Z"}
{"log":"\u0009\u0009 bucket size:       128 -       128 (first       128)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403464386Z"}

{"log":"/usr/sbin/opensips[14]: hash =  30 fragments no.:     2, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403501568Z"}
{"log":"\u0009\u0009 bucket size:       240 -       240 (first       240)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403526919Z"}

{"log":"/usr/sbin/opensips[14]: hash =  43 fragments no.:     2, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403581198Z"}
{"log":"\u0009\u0009 bucket size:       344 -       344 (first       344)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403618498Z"}

{"log":"/usr/sbin/opensips[14]: hash = 114 fragments no.:     2, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403738881Z"}
{"log":"\u0009\u0009 bucket size:       912 -       912 (first       912)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403790258Z"}

{"log":"/usr/sbin/opensips[14]: hash = 309 fragments no.:     1, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403898798Z"}
{"log":"\u0009\u0009 bucket size:      2472 -      2472 (first      2472)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.40393292Z"}

{"log":"/usr/sbin/opensips[14]: hash = 2060 fragments no.:     1, unused:     0\r\n","stream":"stdout","time":"2024-07-09T08:33:07.403982094Z"}
{"log":"\u0009\u0009 bucket size:  33554432 -  67108864 (first  44956816)\r\n","stream":"stdout","time":"2024-07-09T08:33:07.40400207Z"}

{"log":"/usr/sbin/opensips[14]: TOTAL:  10766 free fragments = 45222192 free bytes\r\n","stream":"stdout","time":"2024-07-09T08:33:07.404041247Z"}

{"log":"/usr/sbin/opensips[14]: TOTAL: 56 overhead\r\n","stream":"stdout","time":"2024-07-09T08:33:07.404083739Z"}

{"log":"/usr/sbin/opensips[14]: -----------------------------\r\n","stream":"stdout","time":"2024-07-09T08:33:07.404186485Z"}

Graph of fragments increase and memory usage

Opensips all processes fragments increase: fragments increase

Memory usage graph for (pid:14 id:8): memoryusage

OS/environment information

Additional Info: This only happens with outbound calls. For now I am only taking inbound calls to this sbc and opensips working stable. Fragments counts and memory is stable.

There are two interface for calls, private and public. Problem happens if i enable outbound calls, pids with type: SIP receiver udp:public ip:5060 starts to grow every second.

{
            "ID": 8,
            "PID": 14,
            "Type": "SIP receiver udp:xxx.xxx.xxx.xxx:5060"
},
github-actions[bot] commented 1 month ago

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

musicianx2 commented 1 month ago

Found the related problem #3233 and solved with updating Opensips to version 3.4