search

sipwise / rtpengine

The Sipwise media proxy for Kamailio
GNU General Public License v3.0
784 stars 368 forks source link

Crash in function proc_blist_read (kernel mode). #1797

Closed YuriyFateev closed 7 months ago

YuriyFateev commented 7 months ago

rtpengine version the issue has been seen with

11.5.1.3+0~mr11.5.1.3 git-HEAD-185efb6

Used distribution and its version

Debian 11

Linux kernel version used

5.10.0-26-cloud-amd64

CPU architecture issue was seen on (see uname -m)

x86_64

Expected behaviour you didn't see

No response

Unexpected behaviour you saw

This leads to the fact that after a failure, calls are dropped after 1 minute.

Steps to reproduce the problem

Crashes happen randomly.

Additional program output to the terminal or logs illustrating the issue

kernel stats: page allocation failure: order:4, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 1 PID: 2521 Comm: kernel stats Tainted: G           OE     5.10.0-26-cloud-amd64 #1 Debian 5.10.197-1
Hardware name: Amazon EC2 t3a.nano/, BIOS 1.0 10/16/2017
Call Trace:
 dump_stack+0x6b/0x83
 warn_alloc+0x138/0x160
 ? __alloc_pages_direct_compact+0x167/0x170
 __alloc_pages_slowpath.constprop.0+0xc54/0xc90
 __alloc_pages_nodemask+0x2de/0x310
 kmalloc_order+0x28/0x100
 kmalloc_order_trace+0x19/0x90
 proc_blist_read+0x20f/0x480 [xt_RTPENGINE]
 ? apparmor_file_permission+0x69/0x160
 proc_reg_read+0x51/0x90
 vfs_read+0x98/0x180
 ksys_read+0x5f/0xe0
 do_syscall_64+0x33/0x40
 entry_SYSCALL_64_after_hwframe+0x62/0xc7
RIP: 0033:0x7fd59808708c
Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 89 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 bf fc ff ff 48
RSP: 002b:00007fd59559b5e0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fd59808708c
RDX: 000000000000aa28 RSI: 00007fd55400f000 RDI: 0000000000000015
RBP: 00007fd55400f000 R08: 0000000000000001 R09: 00007fd554000080
R10: 0000000002000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000015 R14: 0000000000030d40 R15: 00007fd59559b780
Mem-Info:
active_anon:21457 inactive_anon:46464 isolated_anon:0
 active_file:18070 inactive_file:7792 isolated_file:0
 unevictable:0 dirty:160 writeback:0
 slab_reclaimable:4602 slab_unreclaimable:6710
 mapped:19769 shmem:6639 pagetables:2159 bounce:0
 free:1936 free_pcp:364 free_cma:0
Node 0 active_anon:85828kB inactive_anon:185856kB active_file:72280kB inactive_file:31168kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:79076kB dirty:640kB writeback:0kB shmem:26556kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:4688kB all_unreclaimable? no
Node 0 DMA free:1820kB min:156kB low:192kB high:228kB reserved_highatomic:0KB active_anon:4320kB inactive_anon:5468kB active_file:1736kB inactive_file:1248kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB pagetables:412kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 417 417 417 417
Node 0 DMA32 free:5924kB min:6552kB low:7676kB high:8800kB reserved_highatomic:0KB active_anon:81636kB inactive_anon:179812kB active_file:70756kB inactive_file:30432kB unevictable:0kB writepending:640kB present:487336kB managed:450588kB mlocked:0kB pagetables:8224kB bounce:0kB free_pcp:1456kB local_pcp:720kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0 0
Node 0 DMA: 80*4kB (ME) 75*8kB (UME) 20*16kB (UME) 10*32kB (E) 4*64kB (E) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1816kB
Node 0 DMA32: 623*4kB (UME) 190*8kB (UM) 79*16kB (UM) 17*32kB (UME) 4*64kB (ME) 1*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6460kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
36808 total pagecache pages
4381 pages in swap cache
Swap cache stats: add 60091, delete 55713, find 466437/473551
Free swap  = 338172kB
Total swap = 524284kB
125832 pages RAM
0 pages HighMem/MovableOnly
9208 pages reserved
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
PGD 0 P4D 0 
Oops: 0002 [#1] SMP NOPTI
CPU: 1 PID: 2521 Comm: kernel stats Tainted: G           OE     5.10.0-26-cloud-amd64 #1 Debian 5.10.197-1
Hardware name: Amazon EC2 t3a.nano/, BIOS 1.0 10/16/2017
RIP: 0010:proc_blist_read+0x225/0x480 [xt_RTPENGINE]
Code: aa ff ff be c0 0d 00 00 bf 28 aa 00 00 89 c2 e8 d1 94 29 f1 48 8d 75 08 b9 5e 00 00 00 48 8d 95 40 03 00 00 48 89 c7 49 89 c7 <f3> 48 a5 48 8b 85 00 03 00 00 49 89 87 f0 02 00 00 48 8b 85 08 03
RSP: 0018:ffffabccc088fe68 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000002 RCX: 000000000000005e
RDX: ffff983a05688b40 RSI: ffff983a05688808 RDI: 0000000000000000
RBP: ffff983a05688800 R08: 0000000000000000 R09: ffffabccc088fa60
R10: 0000000000000002 R11: ffffffffb2ec96e8 R12: ffff983a1103c000
R13: ffffabccc088ff10 R14: 0000000000000094 R15: 0000000000000000
FS:  00007fd59559c700(0000) GS:ffff983a1e100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000001d1e000 CR4: 00000000003506a0
Call Trace:
 ? __die_body.cold+0x1a/0x1f
 ? no_context+0x1a6/0x3c0
 ? exc_page_fault+0x78/0x160
 ? asm_exc_page_fault+0x1e/0x30
 ? proc_blist_read+0x225/0x480 [xt_RTPENGINE]
 ? proc_blist_read+0x20f/0x480 [xt_RTPENGINE]
 ? apparmor_file_permission+0x69/0x160
 proc_reg_read+0x51/0x90
 vfs_read+0x98/0x180
 ksys_read+0x5f/0xe0
 do_syscall_64+0x33/0x40
 entry_SYSCALL_64_after_hwframe+0x62/0xc7
RIP: 0033:0x7fd59808708c
Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 89 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 bf fc ff ff 48
RSP: 002b:00007fd59559b5e0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fd59808708c
RDX: 000000000000aa28 RSI: 00007fd55400f000 RDI: 0000000000000015
RBP: 00007fd55400f000 R08: 0000000000000001 R09: 00007fd554000080
R10: 0000000002000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000015 R14: 0000000000030d40 R15: 00007fd59559b780
Modules linked in: xt_comment sctp iptable_filter xt_RTPENGINE(OE) nft_chain_nat xt_REDIRECT nf_nat xt_state xt_recent xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp dummy xt_multiport ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common nft_limit xt_LOG xt_limit nft_counter xt_set nft_compat nf_tables libcrc32c ip_set_hash_net ip_set nfnetlink crct10dif_pclmul crc32_pclmul ghash_clmulni_intel nls_ascii nls_cp437 aesni_intel vfat fat crypto_simd cryptd glue_helper evdev serio_raw button sunrpc fuse configfs ip_tables x_tables autofs4 crc32c_intel ena
CR2: 0000000000000000
---[ end trace c665ca808024ac01 ]---
RIP: 0010:proc_blist_read+0x225/0x480 [xt_RTPENGINE]
Code: aa ff ff be c0 0d 00 00 bf 28 aa 00 00 89 c2 e8 d1 94 29 f1 48 8d 75 08 b9 5e 00 00 00 48 8d 95 40 03 00 00 48 89 c7 49 89 c7 <f3> 48 a5 48 8b 85 00 03 00 00 49 89 87 f0 02 00 00 48 8b 85 08 03
RSP: 0018:ffffabccc088fe68 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000002 RCX: 000000000000005e
RDX: ffff983a05688b40 RSI: ffff983a05688808 RDI: 0000000000000000
RBP: ffff983a05688800 R08: 0000000000000000 R09: ffffabccc088fa60
R10: 0000000000000002 R11: ffffffffb2ec96e8 R12: ffff983a1103c000
R13: ffffabccc088ff10 R14: 0000000000000094 R15: 0000000000000000
FS:  00007fd59559c700(0000) GS:ffff983a1e100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000001d1e000 CR4: 00000000003506a0

Anything else?

Works well on version rtpengine 9.5.1.4+0~mr9.5.1.4 git-HEAD-32447b0

rfuchs commented 7 months ago

Looks like you're just out of memory, isn't it?

YuriyFateev commented 7 months ago

Thank you very much, it helped