Closed RomanBelozerov closed 11 months ago
krizhanovsky
commented
1 year ago It seems we have some bug, but the very first kernel message seems about out of memory, so I'm wondering if larger memory on a testing VM removes at least the first kernel message?
RomanBelozerov
commented
1 year ago It seems we have some bug, but the very first kernel message seems about out of memory, so I'm wondering if larger memory on a testing VM removes at least the first kernel message?
I received kernel panic when memory VM - 12 GB (it max for my PC). And also I have trace in dmesg (without kernel panic):
[ 288.842334] curl: page allocation failure: order:2, mode:0xa20(GFP_ATOMIC), nodemask=(null),cpuset=user.slice,mems_allowed=0
[ 288.844287] CPU: 2 PID: 15549 Comm: curl Tainted: G OE 5.10.35.tfw-4c9ba16 #1
[ 288.845204] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1 04/01/2014
[ 288.846378] Call Trace:
[ 288.846944] <IRQ>
[ 288.847485] dump_stack+0x74/0x92
[ 288.848101] warn_alloc.cold+0x7b/0xdf
[ 288.848740] __alloc_pages_slowpath.constprop.0+0xd2e/0xd60
[ 288.849511] __alloc_pages_nodemask+0x2cf/0x330
[ 288.850211] alloc_pages_current+0x87/0xe0
[ 288.850884] ss_skb_to_sgvec_with_new_pages+0x91/0x290 [tempesta_fw]
[ 288.851762] tfw_tls_encrypt+0x38e/0x8a0 [tempesta_fw]
[ 288.852507] ? memcpy_fast+0xe/0x10 [tempesta_lib]
[ 288.853256] ? tfw_strcpy+0x1ae/0x2b0 [tempesta_fw]
[ 288.853969] ? tfw_h2_insert_frame_header+0x19b/0x1c0 [tempesta_fw]
[ 288.854806] ? tso_fragment+0x149/0x200
[ 288.855473] ? tfw_sk_prepare_xmit+0x358/0x780 [tempesta_fw]
[ 288.856244] tfw_sk_write_xmit+0x6a/0xc0 [tempesta_fw]
[ 288.856963] tcp_tfw_sk_write_xmit+0x36/0x80
[ 288.857617] tcp_write_xmit+0x2a9/0x11b0
[ 288.858267] __tcp_push_pending_frames+0x37/0x100
[ 288.858965] tcp_rcv_established+0x26c/0x670
[ 288.859636] ? sk_filter_trim_cap+0xde/0x240
[ 288.860303] tcp_v4_do_rcv+0x140/0x200
[ 288.860914] tcp_v4_rcv+0xcfd/0xe10
[ 288.861497] ip_protocol_deliver_rcu+0x30/0x1b0
[ 288.862152] ip_local_deliver_finish+0x48/0x60
[ 288.862809] ip_local_deliver+0xfa/0x110
[ 288.863436] ? ip_protocol_deliver_rcu+0x1b0/0x1b0
[ 288.864094] ip_rcv_finish+0x87/0xa0
[ 288.864663] ip_rcv+0xcc/0xe0
[ 288.865189] ? ip_rcv_finish_core.isra.0+0x420/0x420
[ 288.865849] __netif_receive_skb_one_core+0x88/0xa0
[ 288.866520] __netif_receive_skb+0x18/0x60
[ 288.867158] process_backlog+0xa9/0x160
[ 288.867732] net_rx_action+0x13e/0x390
[ 288.868284] __do_softirq+0xd9/0x291
[ 288.868826] asm_call_irq_on_stack+0xf/0x20
[ 288.869405] </IRQ>
[ 288.869829] do_softirq_own_stack+0x3d/0x50
[ 288.870409] do_softirq.part.0+0x46/0x50
[ 288.870987] __local_bh_enable_ip+0x50/0x60
[ 288.871613] ip_finish_output2+0x1ab/0x590
[ 288.872197] ? __cgroup_bpf_run_filter_skb+0x3c3/0x3d0
[ 288.872861] __ip_finish_output+0xd8/0x220
[ 288.873440] ip_finish_output+0x2d/0xb0
[ 288.873999] ip_output+0x7a/0x100
[ 288.874513] ? __ip_finish_output+0x220/0x220
[ 288.875108] ip_local_out+0x3d/0x50
[ 288.875616] __ip_queue_xmit+0x17a/0x470
[ 288.876145] ? _copy_to_iter+0xa0/0x460
[ 288.876658] ip_queue_xmit+0x15/0x20
[ 288.877160] __tcp_transmit_skb+0xa20/0xbe0
[ 288.877697] __tcp_send_ack.part.0+0xcb/0x1a0
[ 288.878253] tcp_send_ack+0x1c/0x20
[ 288.878745] tcp_cleanup_rbuf+0x7c/0x100
[ 288.879286] tcp_recvmsg+0x258/0xae0
[ 288.879781] inet_recvmsg+0x5f/0x110
[ 288.880270] sock_recvmsg+0x69/0x80
[ 288.880749] sock_read_iter+0x8f/0xf0
[ 288.881249] new_sync_read+0x183/0x1a0
[ 288.881741] vfs_read+0xfe/0x190
[ 288.882195] ksys_read+0xb1/0xe0
[ 288.882641] __x64_sys_read+0x1a/0x20
[ 288.883129] do_syscall_64+0x38/0x90
[ 288.883602] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 288.884169] RIP: 0033:0x7f3dc8098992
[ 288.884636] Code: c0 e9 b2 fe ff ff 50 48 8d 3d fa b2 0c 00 e8 c5 1d 02 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 48 83 ec 28 48 89 54 24
[ 288.886416] RSP: 002b:00007ffd8277d108 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 288.887212] RAX: ffffffffffffffda RBX: 0000000000002d61 RCX: 00007f3dc8098992
[ 288.887959] RDX: 0000000000002d61 RSI: 000055f604019338 RDI: 000000000000001c
[ 288.888716] RBP: 000055f60401f290 R08: 0000000000000001 R09: 00007ffd8277d2e8
[ 288.889466] R10: 0000000000000000 R11: 0000000000000246 R12: 000055f604019338
[ 288.890220] R13: 00007f3dc7a6f6c0 R14: 000055f60414acf0 R15: 00007ffd8277d2e8
[ 288.890998] Mem-Info:
[ 288.891497] active_anon:363 inactive_anon:232010 isolated_anon:0
active_file:38678 inactive_file:32937 isolated_file:0
unevictable:0 dirty:81 writeback:0
slab_reclaimable:11626 slab_unreclaimable:16153
mapped:44570 shmem:5431 pagetables:3054 bounce:0
free:726393 free_pcp:446 free_cma:0
Scope
Tempesta commit - ff0f02145172b92794ef3cd086a12af737caa793
config (all servoces on one VM):
VM memory - 8GB.
First kernel panic:
and second:
Testing
t_stress.test_wordpress.H2WordpressStress.test_get_large_imageswith -R 5 and local config (all services on one VM)