search

tempesta-tech / tempesta

All-in-one solution for high performance web content delivery and advanced protection against DDoS and web attacks
https://tempesta-tech.com/
GNU General Public License v2.0
621 stars 103 forks source link

A lot of BUGS on stress tests with MTU80 #2034

Closed EvgeniiMekhanik closed 9 months ago

EvgeniiMekhanik commented 10 months ago

First BUG

[ 660.912387] [tempesta fw] Tempesta FW is ready [ 676.112430] [tempesta fw] Warning: pending active connections for 5s (connections count 0x1, queues count 0) [ 676.113060] [tempesta fw] Warning: cpu 0(4), backlog size 0, active connections mask 0xfffffffb00000000, cntwork queue size 0, close backlog is empty [ 676.113880] [tempesta fw] Warning: cpu 1(4), backlog size 0, active connections mask 0x200000000, cntwork queue size 0, close backlog is empty [ 676.114660] [tempesta fw] Warning: cpu 2(4), backlog size 0, active connections mask 0xffffffe900000000, cntwork queue size 0, close backlog is empty [ 676.115455] [tempesta fw] Warning: cpu 3(4), backlog size 0, active connections mask 0x300000000, cntwork queue size 0, close backlog is empty [ 676.116221] [tempesta fw] Warning: cpu 4(4), backlog size 0, active connections mask 0x1800000000, cntwork queue size 0, close backlog is empty [ 676.117015] [tempesta fw] Warning: cpu 5(4), backlog size 0, active connections mask 0x1300000000, cntwork queue size 0, close backlog is empty [ 676.117898] [tempesta fw] Warning: cpu 6(4), backlog size 0, active connections mask 0xfffffff700000000, cntwork queue size 0, close backlog is empty [ 676.118805] [tempesta fw] Warning: cpu 7(4), backlog size 0, active connections mask 0xfffffff600000000, cntwork queue size 0, close backlog is empty [ 681.168547] [tempesta fw] Warning: pending active connections for 5s (connections count 0x1, queues count 0) [ 681.169251] [tempesta fw] Warning: cpu 0(4), backlog size 0, active connections mask 0xfffffffb00000000, cntwork queue size 0, close backlog is empty [ 681.170088] [tempesta fw] Warning: cpu 1(4), backlog size 0, active connections mask 0x200000000, cntwork queue size 0, close backlog is empty [ 681.170891] [tempesta fw] Warning: cpu 2(4), backlog size 0, active connections mask 0xffffffe900000000, cntwork queue size 0, close backlog is empty [ 681.171728] [tempesta fw] Warning: cpu 3(4), backlog size 0, active connections mask 0x300000000, cntwork queue size 0, close backlog is empty [ 681.172527] [tempesta fw] Warning: cpu 4(4), backlog size 0, active connections mask 0x1800000000, cntwork queue size 0, close backlog is empty [ 681.173343] [tempesta fw] Warning: cpu 5(4), backlog size 0, active connections mask 0x1300000000, cntwork queue size 0, close backlog is empty [ 681.174288] [tempesta fw] Warning: cpu 6(4), backlog size 0, active connections mask 0xfffffff700000000, cntwork queue size 0, close backlog is empty [ 681.175159] [tempesta fw] Warning: cpu 7(4), backlog size 0, active connections mask 0xfffffff600000000, cntwork queue size 0, close backlog is empty [ 686.176649] [tempesta fw] Warning: pending active connections for 5s (connections count 0x1, queues count 0) [ 686.177298] [tempesta fw] Warning: cpu 0(4), backlog size 0, active connections mask 0xfffffffb00000000, cntwork queue size 0, close backlog is empty [ 686.178133] [tempesta fw] Warning: cpu 1(4), backlog size 0, active connections mask 0x200000000, cntwork queue size 0, close backlog is empty [ 686.178932] [tempesta fw] Warning: cpu 2(4), backlog size 0, active connections mask 0xffffffe900000000, cntwork queue size 0, close backlog is empty [ 686.179767] [tempesta fw] Warning: cpu 3(4), backlog size 0, active connections mask 0x300000000, cntwork queue size 0, close backlog is empty [ 686.180591] [tempesta fw] Warning: cpu 4(4), backlog size 0, active connections mask 0x1800000000, cntwork queue size 0, close backlog is empty [ 686.181408] [tempesta fw] Warning: cpu 5(4), backlog size 0, active connections mask 0x1300000000, cntwork queue size 0, close backlog is empty [ 686.182337] [tempesta fw] Warning: cpu 6(4), backlog size 0, active connections mask 0xfffffff700000000, cntwork queue size 0, close backlog is empty [ 686.183277] [tempesta fw] Warning: cpu 7(4), backlog size 0, active connections mask 0xfffffff600000000, cntwork queue size 0, close backlog is empty [ 686.184235] ------------[ cut here ]------------ [ 686.184562] WARNING: CPU: 4 PID: 15551 at /home/evgeny/workdir/tempesta/fw/main.c:172 tfw_mods_stop+0xa4/0xc0 [tempesta_fw] [ 686.185252] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) tls sha256_ssse3 sha512_ssse3 xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc overlay binfmt_misc nls_iso8859_1 virtio_gpu kvm_intel virtio_dma_buf drm_kms_helper kvm cec rc_core fb_sys_fops crct10dif_pclmul syscopyarea ghash_clmulni_intel sysfillrect sysimgblt aesni_intel crypto_simd joydev cryptd input_leds glue_helper mac_hid qemu_fw_cfg serio_raw sch_fq_codel msr drm parport_pc ppdev lp pstore_blk parport ramoops pstore_zone reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c psmouse e1000 crc32_pclmul i2c_piix4 floppy pata_acpi [last unloaded: tempesta_lib] [ 686.190082] CPU: 4 PID: 15551 Comm: sysctl Tainted: G OE 5.10.35+ #273 [ 686.190563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 686.191089] RIP: 0010:tfw_mods_stop+0xa4/0xc0 [tempesta_fw] [ 686.191437] Code: 8b 05 f0 06 07 02 85 c0 75 24 e8 37 2a 11 ef 85 c0 0f 85 29 12 02 00 5b 41 5c 5d c3 e8 55 7f 00 00 e8 c0 58 00 00 84 c0 75 c3 <0f> 0b eb bf 0f 0b 48 c7 c7 10 5a c7 c0 e8 f7 8b 31 ef 66 2e 0f 1f [ 686.192600] RSP: 0018:ffffb37382b8bcf8 EFLAGS: 00010246 [ 686.192931] RAX: 0000000000000000 RBX: ffffffffc0c773e0 RCX: 0000000000000008 [ 686.193400] RDX: 0000000000000008 RSI: 0000000000000000 RDI: ffffffffb81d0460 [ 686.193842] RBP: ffffb37382b8bd08 R08: 0000000000000000 R09: 0000000000000008 [ 686.194285] R10: 662061747365706d R11: 747365706d65745b R12: 0000000000000000 [ 686.194730] R13: ffffb37382b8bd58 R14: ffff9802f0645428 R15: ffffb37382b8bdc8 [ 686.195179] FS: 00007fe248df7740(0000) GS:ffff9803f7d00000(0000) knlGS:0000000000000000 [ 686.195680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 686.196038] CR2: 000055a7fa54efb0 CR3: 000000015f0fe000 CR4: 0000000000750ee0 [ 686.196481] PKRU: 55555554 [ 686.196658] Call Trace: [ 686.196834] tfw_ctlfn_state_io+0x153/0x3d0 [tempesta_fw] [ 686.197175] ? tfw_cleanup+0x40/0x40 [tempesta_fw] [ 686.197478] proc_sys_call_handler+0x12e/0x240 [ 686.197756] proc_sys_write+0x13/0x20 [ 686.198005] new_sync_write+0x114/0x1b0 [ 686.198246] vfs_write+0x1c0/0x270 [ 686.198461] ksys_write+0x67/0xe0 [ 686.198671] x64_sys_write+0x19/0x20 [ 686.198911] do_syscall_64+0x38/0x90 [ 686.199137] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 686.199452] RIP: 0033:0x7fe248f0ea77 [ 686.199678] Code: 10 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 686.200833] RSP: 002b:00007ffcf1f25558 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 686.201381] RAX: ffffffffffffffda RBX: 000055cf79b60560 RCX: 00007fe248f0ea77 [ 686.201874] RDX: 0000000000000005 RSI: 000055cf79b605a0 RDI: 0000000000000004 [ 686.202356] RBP: 000055cf79b626d0 R08: 0000000000000010 R09: 000055cf79b626d0 [ 686.202843] R10: 0000000000000077 R11: 0000000000000246 R12: 0000000000000005 [ 686.203350] R13: 0000000000000005 R14: 00007fe24900fb80 R15: 00007fe24900fa00 [ 686.203869] ---[ end trace 7fe6ab7f40a6f42b ]--- [ 686.222181] [tdb] Close table 'client0.tdb' [ 686.227359] [tdb] Close table 'sessions0.tdb' [ 686.274046] [tdb] Close table 'filter0.tdb' [ 686.274526] [tempesta fw] modules are stopped [ 691.296749] [tempesta fw] ERROR: Got stuck in releasing of server group objects! 1 objects was not released. [ 691.313097] [tempesta fw] exiting... [ 691.337413] ============================================================================= [ 691.338638] BUG tfw_https_conn_cache (Tainted: G W OE ): Objects remaining in tfw_https_conn_cache on kmem_cache_shutdown() [ 691.340585] ----------------------------------------------------------------------------- [ 691.340585] [ 691.342476] Disabling lock debugging due to kernel taint [ 691.343230] INFO: Slab 0x00000000c27067c5 objects=17 used=3 fp=0x0000000040929c66 flags=0x17ffffc0010200 [ 691.343833] CPU: 3 PID: 15578 Comm: rmmod Tainted: G B W OE 5.10.35+ #273 [ 691.344319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 691.344851] Call Trace: [ 691.345020] dump_stack+0x70/0x8b [ 691.345236] slab_err+0xb7/0xdc [ 691.345442] ? slub_cpu_dead+0xf0/0x120 [ 691.345690] ? slub_cpu_dead+0x120/0x120 [ 691.345943] kmem_cache_shutdown+0x165/0x330 [ 691.346230] kmem_cache_destroy+0x59/0x110 [ 691.346504] tfw_sock_clnt_exit+0x2d/0x40 [tempesta_fw] [ 691.346850] tfw_exit+0xd7/0xea [tempesta_fw] [ 691.347168] __do_sys_delete_module.constprop.0+0x181/0x320 [ 691.347525] ? exit_to_user_mode_prepare+0x35/0x150 [ 691.347838] x64_sys_delete_module+0x12/0x20 [ 691.348125] do_syscall_64+0x38/0x90 [ 691.348358] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 691.348682] RIP: 0033:0x7fbb9d09ecdb [ 691.348915] Code: 73 01 c3 48 8b 0d 55 21 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 25 21 0f 00 f7 d8 64 89 01 48 [ 691.350094] RSP: 002b:00007ffceee4f918 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 691.350575] RAX: ffffffffffffffda RBX: 000055b6079c1770 RCX: 00007fbb9d09ecdb [ 691.351033] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000055b6079c17d8 [ 691.351486] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 691.351939] R10: 00007fbb9d136ac0 R11: 0000000000000206 R12: 00007ffceee4fb70 [ 691.352392] R13: 000055b6079c12a0 R14: 00007ffceee5168b R15: 000055b6079c1770 [ 691.352847] INFO: Object 0x0000000071559e73 @offset=0 [ 691.353251] INFO: Object 0x0000000017fcb81e @offset=22944 [ 691.353596] INFO: Object 0x0000000086b6118b @offset=28680 [ 691.353964] kmem_cache_destroy tfw_https_conn_cache: Slab cache still has objects [ 691.354621] CPU: 3 PID: 15578 Comm: rmmod Tainted: G B W OE 5.10.35+ #273 [ 691.355519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 691.356306] Call Trace: [ 691.356470] dump_stack+0x70/0x8b [ 691.356686] kmem_cache_destroy+0x107/0x110 [ 691.356966] tfw_sock_clnt_exit+0x2d/0x40 [tempesta_fw] [ 691.357313] tfw_exit+0xd7/0xea [tempesta_fw] [ 691.357594] do_sys_delete_module.constprop.0+0x181/0x320 [ 691.357951] ? exit_to_user_mode_prepare+0x35/0x150 [ 691.358264] x64_sys_delete_module+0x12/0x20 [ 691.358549] do_syscall_64+0x38/0x90 [ 691.358781] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 691.359218] RIP: 0033:0x7fbb9d09ecdb [ 691.359450] Code: 73 01 c3 48 8b 0d 55 21 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 25 21 0f 00 f7 d8 64 89 01 48 [ 691.360629] RSP: 002b:00007ffceee4f918 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 691.361113] RAX: ffffffffffffffda RBX: 000055b6079c1770 RCX: 00007fbb9d09ecdb [ 691.361567] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000055b6079c17d8 [ 691.362021] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 691.362484] R10: 00007fbb9d136ac0 R11: 0000000000000206 R12: 00007ffceee4fb70 [ 691.362946] R13: 000055b6079c12a0 R14: 00007ffceee5168b R15: 000055b6079c1770 [ 691.363575] ============================================================================= [ 691.364097] BUG tfw_srv_conn_cache (Tainted: G B W OE ): Objects remaining in tfw_srv_conn_cache on kmem_cache_shutdown() [ 691.364850] ----------------------------------------------------------------------------- [ 691.364850] [ 691.365481] INFO: Slab 0x0000000077147fdd objects=31 used=1 fp=0x000000002a07ea64 flags=0x17ffffc0010200 [ 691.366081] CPU: 3 PID: 15578 Comm: rmmod Tainted: G B W OE 5.10.35+ #273 [ 691.366567] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 691.367099] Call Trace: [ 691.367261] dump_stack+0x70/0x8b [ 691.367476] slab_err+0xb7/0xdc [ 691.367681] ? slub_cpu_dead+0x120/0x120 [ 691.367934] ? slub_cpu_dead+0x120/0x120 [ 691.368187] ? on_each_cpu_cond_mask+0x48/0x90 [ 691.368472] kmem_cache_shutdown+0x165/0x330 [ 691.368757] kmem_cache_destroy+0x59/0x110 [ 691.369054] tfw_sock_srv_exit+0x21/0x30 [tempesta_fw] [ 691.369391] tfw_exit+0xd7/0xea [tempesta_fw] [ 691.369673] do_sys_delete_module.constprop.0+0x181/0x320 [ 691.370032] ? exit_to_user_mode_prepare+0x35/0x150 [ 691.370346] x64_sys_delete_module+0x12/0x20 [ 691.370634] do_syscall_64+0x38/0x90 [ 691.370871] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 691.371210] RIP: 0033:0x7fbb9d09ecdb [ 691.371442] Code: 73 01 c3 48 8b 0d 55 21 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 25 21 0f 00 f7 d8 64 89 01 48 [ 691.372628] RSP: 002b:00007ffceee4f918 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 691.373114] RAX: ffffffffffffffda RBX: 000055b6079c1770 RCX: 00007fbb9d09ecdb [ 691.373570] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000055b6079c17d8 [ 691.374025] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 691.374481] R10: 00007fbb9d136ac0 R11: 0000000000000206 R12: 00007ffceee4fb70 [ 691.374938] R13: 000055b6079c12a0 R14: 00007ffceee5168b R15: 000055b6079c1770 [ 691.375395] INFO: Object 0x000000001e825b3b @offset=6288 [ 691.375743] kmem_cache_destroy tfw_srv_conn_cache: Slab cache still has objects [ 691.376244] CPU: 3 PID: 15578 Comm: rmmod Tainted: G B W OE 5.10.35+ #273 [ 691.376885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 691.377492] Call Trace: [ 691.377660] dump_stack+0x70/0x8b [ 691.377877] kmem_cache_destroy+0x107/0x110 [ 691.378160] tfw_sock_srv_exit+0x21/0x30 [tempesta_fw] [ 691.378496] tfw_exit+0xd7/0xea [tempesta_fw] [ 691.378777] do_sys_delete_module.constprop.0+0x181/0x320 [ 691.379168] ? exit_to_user_mode_prepare+0x35/0x150 [ 691.379481] x64_sys_delete_module+0x12/0x20 [ 691.379766] do_syscall_64+0x38/0x90 [ 691.379997] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 691.380320] RIP: 0033:0x7fbb9d09ecdb [ 691.380551] Code: 73 01 c3 48 8b 0d 55 21 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 25 21 0f 00 f7 d8 64 89 01 48 [ 691.381730] RSP: 002b:00007ffceee4f918 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 691.382210] RAX: ffffffffffffffda RBX: 000055b6079c1770 RCX: 00007fbb9d09ecdb [ 691.382661] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000055b6079c17d8 [ 691.383132] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 691.383584] R10: 00007fbb9d136ac0 R11: 0000000000000206 R12: 00007ffceee4fb70 [ 691.384036] R13: 000055b6079c12a0 R14: 00007ffceee5168b R15: 000055b6079c1770 [ 691.384648] ============================================================================= [ 691.385247] BUG tfw_srv_cache (Tainted: G B W OE ): Objects remaining in tfw_srv_cache on kmem_cache_shutdown() [ 691.386025] ----------------------------------------------------------------------------- [ 691.386025] [ 691.386668] INFO: Slab 0x00000000d6638649 objects=23 used=1 fp=0x000000006637014f flags=0x17ffffc0000200 [ 691.387276] CPU: 3 PID: 15578 Comm: rmmod Tainted: G B W OE 5.10.35+ #273 [ 691.387760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 691.388287] Call Trace: [ 691.388449] dump_stack+0x70/0x8b [ 691.388663] slab_err+0xb7/0xdc [ 691.388870] ? slub_cpu_dead+0x120/0x120 [ 691.389140] ? slub_cpu_dead+0x120/0x120 [ 691.389392] ? on_each_cpu_cond_mask+0x48/0x90 [ 691.389677] __kmem_cache_shutdown+0x165/0x330 [ 691.389962] kmem_cache_destroy+0x59/0x110 [ 691.390232] tfw_server_exit+0x15/0x20 [tempesta_fw] [ 691.390556] tfw_exit+0xd7/0xea [tempesta_fw] [ 691.390877] do_sys_delete_module.constprop.0+0x181/0x320 [ 691.391238] ? exit_to_user_mode_prepare+0x35/0x150 [ 691.391550] x64_sys_delete_module+0x12/0x20 [ 691.391836] do_syscall_64+0x38/0x90 [ 691.392067] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 691.392390] RIP: 0033:0x7fbb9d09ecdb [ 691.392620] Code: 73 01 c3 48 8b 0d 55 21 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 25 21 0f 00 f7 d8 64 89 01 48 [ 691.393802] RSP: 002b:00007ffceee4f918 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 691.394282] RAX: ffffffffffffffda RBX: 000055b6079c1770 RCX: 00007fbb9d09ecdb [ 691.394735] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000055b6079c17d8 [ 691.395209] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 691.395661] R10: 00007fbb9d136ac0 R11: 0000000000000206 R12: 00007ffceee4fb70 [ 691.396113] R13: 000055b6079c12a0 R14: 00007ffceee5168b R15: 000055b6079c1770 [ 691.396567] INFO: Object 0x00000000aeb80e4c @offset=528 [ 691.396913] kmem_cache_destroy tfw_srv_cache: Slab cache still has objects [ 691.397409] CPU: 3 PID: 15578 Comm: rmmod Tainted: G B W OE 5.10.35+ #273 [ 691.397924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 691.398518] Call Trace: [ 691.398689] dump_stack+0x70/0x8b [ 691.398930] kmem_cache_destroy+0x107/0x110 [ 691.399236] tfw_server_exit+0x15/0x20 [tempesta_fw] [ 691.399561] tfw_exit+0xd7/0xea [tempesta_fw] [ 691.399841] do_sys_delete_module.constprop.0+0x181/0x320 [ 691.400198] ? exit_to_user_mode_prepare+0x35/0x150 [ 691.400511] x64_sys_delete_module+0x12/0x20 [ 691.400812] do_syscall_64+0x38/0x90 [ 691.401061] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 691.401387] RIP: 0033:0x7fbb9d09ecdb [ 691.401619] Code: 73 01 c3 48 8b 0d 55 21 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 25 21 0f 00 f7 d8 64 89 01 48 [ 691.402796] RSP: 002b:00007ffceee4f918 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 691.403280] RAX: ffffffffffffffda RBX: 000055b6079c1770 RCX: 00007fbb9d09ecdb [ 691.403732] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000055b6079c17d8 [ 691.404183] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 691.404635] R10: 00007fbb9d136ac0 R11: 0000000000000206 R12: 00007ffceee4fb70 [ 691.405088] R13: 000055b6079c12a0 R14: 00007ffceee5168b R15: 000055b6079c1770 [ 691.477177] [tdb] Shutdown Tempesta DB [ 691.500889] BUG: unable to handle page fault for address: ffffffffc0b98c00 [ 691.502041] #PF: supervisor instruction fetch in kernel mode [ 691.503011] #PF: error_code(0x0010) - not-present page [ 691.503699] PGD 125215067 P4D 125215067 PUD 125217067 PMD 132559067 PTE 0 [ 691.504574] Oops: 0010 [#1] SMP NOPTI [ 691.505048] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G B W OE 5.10.35+ #273 [ 691.505969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 691.506896] RIP: 0010:0xffffffffc0b98c00 [ 691.507299] Code: Unable to access opcode bytes at RIP 0xffffffffc0b98bd6. [ 691.507978] RSP: 0018:ffffb373801c8ec0 EFLAGS: 00010206 [ 691.508433] RAX: 0000000000000001 RBX: 0000000000000100 RCX: 0000000000000240 [ 691.509033] RDX: 0000000100017e33 RSI: ffffffffc0b98c00 RDI: ffff980319cbb080 [ 691.509584] RBP: ffffb373801c8ee0 R08: ffff980319cbb080 R09: 0000000000000201 [ 691.510120] R10: 0000000000000001 R11: ffff9803f7d6c1b0 R12: ffff980319cbb080 [ 691.510672] R13: ffffffffc0b98c00 R14: 0000000000000000 R15: ffff9803f7d5cc80 [ 691.511236] FS: 0000000000000000(0000) GS:ffff9803f7d40000(0000) knlGS:0000000000000000 [ 691.511893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 691.512337] CR2: ffffffffc0b98bd6 CR3: 000000016533a000 CR4: 0000000000750ee0 [ 691.512894] PKRU: 55555554 [ 691.513101] Call Trace: [ 691.513298] [ 691.513474] ? call_timer_fn+0x2e/0x120 [ 691.513766] __run_timers+0x1e6/0x260 [ 691.514064] ? lapic_next_deadline+0x2c/0x40 [ 691.514386] ? clockevents_program_event+0xa9/0x120 [ 691.514983] run_timer_softirq+0x1d/0x40 [ 691.515297] do_softirq+0xe3/0x340 [ 691.515585] asm_call_irq_on_stack+0xf/0x20 [ 691.515903] [ 691.516076] do_softirq_own_stack+0x3d/0x50 [ 691.516423] irq_exit_rcu+0xa2/0xe0 [ 691.516705] sysvec_apic_timer_interrupt+0x3d/0x90 [ 691.517067] asm_sysvec_apic_timer_interrupt+0x12/0x20 [ 691.517482] RIP: 0010:native_safe_halt+0xe/0x10 [ 691.517802] Code: 39 ff ff ff 4c 89 ee 48 c7 c7 00 bb e5 b0 e8 c9 60 91 ff e9 01 ff ff ff cc cc cc cc e9 07 00 00 00 0f 00 2d 26 47 47 00 fb f4 90 e9 07 00 00 00 0f 00 2d 16 47 47 00 f4 c3 cc cc 0f 1f 44 00 [ 691.519229] RSP: 0018:ffffb373800abe88 EFLAGS: 00000216 [ 691.519638] RAX: ffffffffaff97700 RBX: 0000000000000005 RCX: ffff9803f7d6ce00 [ 691.520184] RDX: 0000000000843b8e RSI: ffffb373800abe28 RDI: 000000a1000367b0 [ 691.520705] RBP: ffffb373800abe90 R08: 0000000000000001 R09: 000000000000015a [ 691.521274] R10: 0000000000000003 R11: 0000000000000000 R12: ffff9802e02e3000 [ 691.521865] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 691.522408] ? __sched_text_end+0x4/0x4 [ 691.522710] ? default_idle+0xe/0x20 [ 691.522996] arch_cpu_idle+0x15/0x20 [ 691.523274] default_idle_call+0x3d/0xc0 [ 691.523576] do_idle+0x215/0x2a0 [ 691.523834] cpu_startup_entry+0x20/0x30 [ 691.524152] start_secondary+0x145/0x1b0 [ 691.524453] secondary_startup_64_no_verify+0xc2/0xcb [ 691.524843] Modules linked in: tempesta_lib(OE) tls sha256_ssse3 sha512_ssse3 xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc overlay binfmt_misc nls_iso8859_1 virtio_gpu kvm_intel virtio_dma_buf drm_kms_helper kvm cec rc_core fb_sys_fops crct10dif_pclmul syscopyarea ghash_clmulni_intel sysfillrect sysimgblt aesni_intel crypto_simd joydev cryptd input_leds glue_helper mac_hid qemu_fw_cfg serio_raw sch_fq_codel msr drm parport_pc ppdev lp pstore_blk parport ramoops pstore_zone reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c psmouse e1000 crc32_pclmul i2c_piix4 floppy pata_acpi [last unloaded: tempesta_tls] [ 691.529567] CR2: ffffffffc0b98c00 [ 691.529785] ---[ end trace 7fe6ab7f40a6f42c ]--- [ 691.530082] RIP: 0010:0xffffffffc0b98c00 [ 691.530458] Code: Unable to access opcode bytes at RIP 0xffffffffc0b98bd6. [ 691.530902] RSP: 0018:ffffb373801c8ec0 EFLAGS: 00010206 [ 691.531271] RAX: 0000000000000001 RBX: 0000000000000100 RCX: 0000000000000240 [ 691.531728] RDX: 0000000100017e33 RSI: ffffffffc0b98c00 RDI: ffff980319cbb080 [ 691.532182] RBP: ffffb373801c8ee0 R08: ffff980319cbb080 R09: 0000000000000201 [ 691.532635] R10: 0000000000000001 R11: ffff9803f7d6c1b0 R12: ffff980319cbb080 [ 691.533115] R13: ffffffffc0b98c00 R14: 0000000000000000 R15: ffff9803f7d5cc80 [ 691.533569] FS: 0000000000000000(0000) GS:ffff9803f7d40000(0000) knlGS:0000000000000000 [ 691.534096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 691.534462] CR2: ffffffffc0b98bd6 CR3: 000000016533a000 CR4: 0000000000750ee0 [ 691.534930] PKRU: 55555554 [ 691.535112] Kernel panic - not syncing: Fatal exception in interrupt [ 691.535885] Kernel Offset: 0x2e200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 691.536574] Rebooting in 1 seconds..

Testing

  1. Remove t_stress.test_stress.WrkStressMTU80 t_stress.test_stress.TlsWrkStressMTU80 t_stress.test_stress.H2LoadStressMTU80 from disabled tests.
  2. Run this tests. Run tls/ tests. Run this tests again.
EvgeniiMekhanik commented 10 months ago

Second bug

[ 2480.025590] kernel BUG at /home/evgeny/workdir/tempesta/fw/client.c:92! [ 2480.026065] invalid opcode: 0000 [#1] SMP NOPTI [ 2480.026391] CPU: 4 PID: 21081 Comm: nginx Tainted: G OE 5.10.35+ #273 [ 2480.026930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 2480.027524] RIP: 0010:tfw_client_put+0xa2/0xc0 [tempesta_fw] [ 2480.027924] Code: e8 73 d9 57 d4 8b 05 0d c5 0d 00 48 03 45 d8 4c 89 e7 48 89 83 58 01 00 00 e8 4e ed 53 d4 66 90 65 48 ff 0d 98 17 43 3f eb a4 <0f> 0b 48 c7 c7 a0 a3 cb c0 e8 7d b6 13 d5 e8 cb 9b 18 d5 66 66 2e [ 2480.029299] RSP: 0018:ffffbb45801947e8 EFLAGS: 00010206 [ 2480.029686] RAX: ffff94ea0d211018 RBX: ffff94ea1d211018 RCX: 00000000800d0008 [ 2480.030210] RDX: 0000000000000001 RSI: ffffffff95af332d RDI: ffff94ea1d211178 [ 2480.030744] RBP: ffffbb4580194810 R08: 0000000000000001 R09: 0000000000000000 [ 2480.031251] R10: 0000000000000001 R11: 00000000001cdf00 R12: ffff94ea1d211178 [ 2480.031755] R13: ffff94eb07555d50 R14: ffff94ead522f020 R15: ffff94ead3d45020 [ 2480.032260] FS: 00007f5106b6a740(0000) GS:ffff94eb37d00000(0000) knlGS:0000000000000000 [ 2480.032828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2480.033245] CR2: 00007f5106d90218 CR3: 0000000127532000 CR4: 0000000000750ee0 [ 2480.033766] PKRU: 55555554 [ 2480.033971] Call Trace: [ 2480.034157] [ 2480.034314] ? sk_free+0x71/0x120 [ 2480.034573] tfw_cli_conn_release+0x55/0x90 [tempesta_fw] [ 2480.034970] tfw_tls_conn_dtor+0x134/0x1b0 [tempesta_fw] [ 2480.035327] tfw_http_conn_msg_free+0x81/0x90 [tempesta_fw] [ 2480.035697] tfw_h2_resp_adjust_fwd+0x6d2/0x760 [tempesta_fw] [ 2480.036075] ? ata_scsi_qc_complete+0x92/0x2d0 [ 2480.036373] tfw_http_resp_cache_cb+0x33/0x2f0 [tempesta_fw] [ 2480.036748] ? tfw_h2_resp_adjust_fwd+0x760/0x760 [tempesta_fw] [ 2480.037182] tfw_cache_process+0x63/0x360 [tempesta_fw] [ 2480.037572] tfw_http_resp_cache+0xf7/0x1a0 [tempesta_fw] [ 2480.037970] tfw_http_msg_process_generic+0x50f/0x6c0 [tempesta_fw] [ 2480.038439] ? get_nohz_timer_target+0x2a/0x140 [ 2480.038743] tfw_http_msg_process+0x3e/0x60 [tempesta_fw] [ 2480.039088] tfw_connection_recv+0x87/0x110 [tempesta_fw] [ 2480.039436] ss_tcp_process_data+0x1e8/0x430 [tempesta_fw] [ 2480.039790] ss_tcp_data_ready+0x44/0xf0 [tempesta_fw] [ 2480.040128] tcp_data_ready+0x2b/0xd0 [ 2480.040362] tcp_rcv_established+0x530/0x6b0 [ 2480.040647] tcp_v4_do_rcv+0x140/0x200 [ 2480.040933] tcp_v4_rcv+0xcd0/0xe20 [ 2480.041208] ip_protocol_deliver_rcu+0x44/0x230 [ 2480.041521] ip_local_deliver_finish+0x48/0x60 [ 2480.041828] ip_local_deliver+0xf8/0x110 [ 2480.042121] ? ip_protocol_deliver_rcu+0x230/0x230 [ 2480.042459] ip_rcv_finish+0x87/0xa0 [ 2480.042707] ip_rcv+0xce/0xe0 [ 2480.042907] ? ip_rcv_finish_core.constprop.0+0x470/0x470 [ 2480.043267] netif_receive_skb_one_core+0x86/0xa0 [ 2480.043588] netif_receive_skb+0x18/0x60 [ 2480.043860] process_backlog+0x9e/0x170 [ 2480.044114] net_rx_action+0x13b/0x430 [ 2480.044366] ? scsi_finish_command+0xea/0x120 [ 2480.044668] __do_softirq+0xe3/0x340 [ 2480.044921] asm_call_irq_on_stack+0xf/0x20 [ 2480.045214] [ 2480.045375] do_softirq_own_stack+0x3d/0x50 [ 2480.045661] do_softirq+0x66/0x80 [ 2480.045886] local_bh_enable_ip+0x50/0x60 [ 2480.046164] ip_finish_output2+0x1bb/0x5c0 [ 2480.046445] ? ip_frag_next+0x175/0x1b0 [ 2480.046706] ip_do_fragment+0x1e9/0x520 [ 2480.047005] ? ip_fragment.constprop.0+0x90/0x90 [ 2480.047773] ip_fragment.constprop.0+0x48/0x90 [ 2480.048549] ip_finish_output+0x11c/0x210 [ 2480.049464] ip_finish_output+0x31/0xc0 [ 2480.050316] ip_output+0x78/0x100 [ 2480.051029] ? ip_finish_output+0x210/0x210 [ 2480.051784] ip_queue_xmit+0x18d/0x440 [ 2480.052517] ip_queue_xmit+0x15/0x20 [ 2480.053207] tcp_transmit_skb+0xae3/0xcd0 [ 2480.053904] tcp_write_xmit+0x3c5/0x1060 [ 2480.054592] tcp_push_pending_frames+0x37/0x100 [ 2480.055291] tcp_sock_set_cork.part.0+0x6c/0x80 [ 2480.055996] tcp_setsockopt+0x6cb/0xe80 [ 2480.056615] sock_common_setsockopt+0x1a/0x20 [ 2480.057306] __sys_setsockopt+0x105/0x1f0 [ 2480.057961] __x64_sys_setsockopt+0x1f/0x30 [ 2480.058661] do_syscall_64+0x38/0x90 [ 2480.059277] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 2480.059968] RIP: 0033:0x7f5106c96cbe [ 2480.060584] Code: 0f 1f 40 00 48 8b 15 71 11 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b1 0f 1f 00 f3 0f 1e fa 49 89 ca b8 36 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 0a c3 66 0f 1f 84 00 00 00 00 00 48 8b 15 39 [ 2480.062251] RSP: 002b:00007ffdba059878 EFLAGS: 00000246 ORIG_RAX: 0000000000000036 [ 2480.063179] RAX: ffffffffffffffda RBX: 00005614faf49330 RCX: 00007f5106c96cbe [ 2480.064037] RDX: 0000000000000003 RSI: 0000000000000006 RDI: 0000000000000014 [ 2480.064918] RBP: 00005614faf24968 R08: 0000000000000004 R09: 000000007fffef04 [ 2480.065859] R10: 00007ffdba059894 R11: 0000000000000246 R12: 00007f5106ae2668 [ 2480.066779] R13: 00005614faf3f7c0 R14: 00005614faf24850 R15: 0000000000000000 [ 2480.067674] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) tls sha256_ssse3 sha512_ssse3 xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc overlay binfmt_misc virtio_gpu virtio_dma_buf nls_iso8859_1 drm_kms_helper kvm_intel cec rc_core kvm sch_fq_codel fb_sys_fops syscopyarea sysfillrect sysimgblt crct10dif_pclmul ghash_clmulni_intel msr joydev input_leds aesni_intel crypto_simd cryptd glue_helper serio_raw mac_hid qemu_fw_cfg parport_pc ppdev drm lp parport pstore_blk ramoops reed_solomon pstore_zone efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c psmouse e1000 crc32_pclmul i2c_piix4 floppy pata_acpi [last unloaded: tempesta_lib] [ 2480.075476] ---[ end trace 261290f8821a869b ]--- [ 2480.076417] RIP: 0010:tfw_client_put+0xa2/0xc0 [tempesta_fw] [ 2480.077337] Code: e8 73 d9 57 d4 8b 05 0d c5 0d 00 48 03 45 d8 4c 89 e7 48 89 83 58 01 00 00 e8 4e ed 53 d4 66 90 65 48 ff 0d 98 17 43 3f eb a4 <0f> 0b 48 c7 c7 a0 a3 cb c0 e8 7d b6 13 d5 e8 cb 9b 18 d5 66 66 2e [ 2480.079377] RSP: 0018:ffffbb45801947e8 EFLAGS: 00010206 [ 2480.080295] RAX: ffff94ea0d211018 RBX: ffff94ea1d211018 RCX: 00000000800d0008 [ 2480.081349] RDX: 0000000000000001 RSI: ffffffff95af332d RDI: ffff94ea1d211178 [ 2480.082404] RBP: ffffbb4580194810 R08: 0000000000000001 R09: 0000000000000000 [ 2480.083571] R10: 0000000000000001 R11: 00000000001cdf00 R12: ffff94ea1d211178 [ 2480.084641] R13: ffff94eb07555d50 R14: ffff94ead522f020 R15: ffff94ead3d45020 [ 2480.085687] FS: 00007f5106b6a740(0000) GS:ffff94eb37d00000(0000) knlGS:0000000000000000 [ 2480.086949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2480.087886] CR2: 00007f5106d90218 CR3: 0000000127532000 CR4: 0000000000750ee0 [ 2480.088924] PKRU: 55555554 [ 2480.089778] Kernel panic - not syncing: Fatal exception in interrupt [ 2480.091251] Kernel Offset: 0x14000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 2480.092657] Rebooting in 1 seconds..

EvgeniiMekhanik commented 10 months ago

062a2d1a0a614004f1eee7fcc2905d4cb3a744d2 Tempesta master 3a73c78153c7889ff13fe42ec48e3beaeb585542 Tempesta-test master

EvgeniiMekhanik commented 10 months ago

[ 6364.515597] Start test: t_stress.test_stress.H2LoadStressMTU80.test [ 6370.115321] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.116242] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.117218] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.118130] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.119030] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.119990] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.121141] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.122075] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.123016] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.123979] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.124906] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.125851] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.126794] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.127783] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.128766] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.129994] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.130994] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.131954] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.132920] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.133952] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.134901] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.135884] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.136972] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.137997] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.138947] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.139872] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.140816] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.141746] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.142676] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.143638] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.144565] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.145744] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.146663] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.147553] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.148446] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.149371] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.150350] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.151673] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.152556] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.153672] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.154606] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.155513] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.156413] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.157325] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.158230] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.159156] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.160068] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.161139] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.162037] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.162930] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.163828] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.164815] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.165730] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.166597] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.167452] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.168341] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.169404] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.170297] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.171198] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.172125] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.173032] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.173955] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.174844] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.175726] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.176640] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.177745] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.178684] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.179634] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.180548] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.181469] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.182340] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.183230] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.184160] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.185313] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.186202] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.187081] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.187957] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.188848] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.189731] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.190660] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.191527] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.192420] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.193480] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.194362] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.195248] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.196147] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.197012] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.197839] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.198679] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.199534] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.200377] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.201243] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.202093] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.202925] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.203848] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.204723] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.205705] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.206575] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.207439] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.208382] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.209283] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.210179] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.211069] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.211957] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.212847] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.213741] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.214622] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.215511] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.216418] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.217334] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.218226] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.219086] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.220016] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.220929] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.221823] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.222687] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.223567] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.224455] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.225615] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.226529] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.227414] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.228313] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.229217] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.230110] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.231012] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.231910] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.232809] [tempesta fw] Warning: recvmsg bug: TCP sequence gap at seq 72DB4B79 recvnxt 72DB4B85 [ 6370.233740] ------------[ cut here ]------------ [ 6370.234200] kernel BUG at net/ipv4/tcp_input.c:5161! [ 6370.234704] invalid opcode: 0000 [#1] SMP NOPTI [ 6370.235176] CPU: 1 PID: 20 Comm: ksoftirqd/1 Tainted: G OE 5.10.35+ #273 [ 6370.235975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 6370.236836] RIP: 0010:tcp_collapse+0x527/0x5f0 [ 6370.237288] Code: ff 41 f6 44 24 34 03 0f 85 69 fe ff ff 41 0f b6 84 24 88 00 00 00 41 32 80 88 00 00 00 a8 02 0f 84 52 fd ff ff e9 4c fe ff ff <0f> 0b 48 c7 c7 10 c4 55 8f e8 e8 9b 16 00 48 8b 45 c0 48 8d 7d b8 [ 6370.239142] RSP: 0018:ffffb4e7c00e39d8 EFLAGS: 00010283 [ 6370.239656] RAX: 0000000000000030 RBX: 0000000072db4b79 RCX: 0000000000000000 [ 6370.240354] RDX: 0000000000001280 RSI: ffffa0a8e98ad140 RDI: ffffa0a8e98ac0b8 [ 6370.241098] RBP: ffffb4e7c00e3a48 R08: ffffa0a8e98ac000 R09: 0000000000000ec0 [ 6370.241816] R10: 0000000000001040 R11: 00000000fffffff4 R12: ffffa0a85812fc00 [ 6370.242533] R13: ffffa0a7ed395650 R14: 0000000000000000 R15: ffffa0a7ed395580 [ 6370.243231] FS: 0000000000000000(0000) GS:ffffa0a8f7c40000(0000) knlGS:0000000000000000 [ 6370.244067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6370.244670] CR2: 00007fc4036fb0e8 CR3: 00000001ae2b6000 CR4: 0000000000750ee0 [ 6370.245394] PKRU: 55555554 [ 6370.245677] Call Trace: [ 6370.245940] tcp_try_rmem_schedule+0x2ba/0x420 [ 6370.246369] tcp_data_queue+0x5f3/0xe50 [ 6370.246760] tcp_rcv_established+0x254/0x6b0 [ 6370.247158] tcp_v4_do_rcv+0x140/0x200 [ 6370.247501] tcp_v4_rcv+0xcd0/0xe20 [ 6370.247845] ip_protocol_deliver_rcu+0x44/0x230 [ 6370.248237] ip_local_deliver_finish+0x48/0x60 [ 6370.248602] ip_local_deliver+0xf8/0x110 [ 6370.248928] ? ip_protocol_deliver_rcu+0x230/0x230 [ 6370.249345] ip_rcv_finish+0x87/0xa0 [ 6370.249738] ip_rcv+0xce/0xe0 [ 6370.250057] ? ip_rcv_finish_core.constprop.0+0x470/0x470 [ 6370.250617] netif_receive_skb_one_core+0x86/0xa0 [ 6370.251109] netif_receive_skb+0x18/0x60 [ 6370.251523] process_backlog+0x9e/0x170 [ 6370.251889] net_rx_action+0x13b/0x430 [ 6370.252279] ? invoke_rcu_core+0x40/0xb0 [ 6370.252676] __do_softirq+0xe3/0x340 [ 6370.253053] run_ksoftirqd+0x2b/0x40 [ 6370.253412] smpboot_thread_fn+0xd0/0x190 [ 6370.253817] kthread+0x130/0x150 [ 6370.254136] ? smpboot_register_percpu_thread+0x150/0x150 [ 6370.254672] ? __kthread_bind_mask+0x70/0x70 [ 6370.255098] ret_from_fork+0x1f/0x30 [ 6370.255473] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) tls sha256_ssse3 sha512_ssse3 xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc overlay binfmt_misc nls_iso8859_1 virtio_gpu kvm_intel virtio_dma_buf drm_kms_helper kvm cec rc_core crct10dif_pclmul ghash_clmulni_intel fb_sys_fops aesni_intel syscopyarea sch_fq_codel crypto_simd cryptd glue_helper sysfillrect joydev input_leds sysimgblt serio_raw msr mac_hid qemu_fw_cfg parport_pc ppdev lp drm parport pstore_blk ramoops reed_solomon pstore_zone efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c psmouse e1000 crc32_pclmul floppy i2c_piix4 pata_acpi [last unloaded: tempesta_lib] [ 6370.263205] ---[ end trace 09ae24757a704403 ]--- [ 6370.263656] RIP: 0010:tcp_collapse+0x527/0x5f0 [ 6370.264076] Code: ff 41 f6 44 24 34 03 0f 85 69 fe ff ff 41 0f b6 84 24 88 00 00 00 41 32 80 88 00 00 00 a8 02 0f 84 52 fd ff ff e9 4c fe ff ff <0f> 0b 48 c7 c7 10 c4 55 8f e8 e8 9b 16 00 48 8b 45 c0 48 8d 7d b8 [ 6370.265823] RSP: 0018:ffffb4e7c00e39d8 EFLAGS: 00010283 [ 6370.266311] RAX: 0000000000000030 RBX: 0000000072db4b79 RCX: 0000000000000000 [ 6370.266971] RDX: 0000000000001280 RSI: ffffa0a8e98ad140 RDI: ffffa0a8e98ac0b8 [ 6370.267639] RBP: ffffb4e7c00e3a48 R08: ffffa0a8e98ac000 R09: 0000000000000ec0 [ 6370.268313] R10: 0000000000001040 R11: 00000000fffffff4 R12: ffffa0a85812fc00 [ 6370.268992] R13: ffffa0a7ed395650 R14: 0000000000000000 R15: ffffa0a7ed395580 [ 6370.269688] FS: 0000000000000000(0000) GS:ffffa0a8f7c40000(0000) knlGS:0000000000000000 [ 6370.270442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6370.271031] CR2: 00007fc4036fb0e8 CR3: 00000001ae2b6000 CR4: 0000000000750ee0 [ 6370.271760] PKRU: 55555554 [ 6370.272022] Kernel panic - not syncing: Fatal exception in interrupt [ 6370.272843] Kernel Offset: 0x5600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 6370.273692] Rebooting in 1 seconds..

EvgeniiMekhanik commented 9 months ago

[ 1768.037843] [tempesta fw] Warning: cpu 0(4), backlog size 0, active connections mask 0x0, cntwork queue size 0, close backlog is empty [ 1768.038586] [tempesta fw] Warning: cpu 1(4), backlog size 0, active connections mask 0x0, cntwork queue size 0, close backlog is empty [ 1768.039332] [tempesta fw] Warning: cpu 2(4), backlog size 0, active connections mask 0x100000000, cntwork queue size 0, close backlog is empty [ 1768.040133] [tempesta fw] Warning: cpu 3(4), backlog size 0, active connections mask 0x0, cntwork queue size 0, close backlog is empty [ 1768.040876] [tempesta fw] Warning: cpu 4(4), backlog size 0, active connections mask 0x2000000000, cntwork queue size 0, close backlog is empty [ 1769.503358] skbuff: skb_under_panic: text:ffffffffc0c68060 len:405 put:13 head:ffff98d66e76e580 data:ffff98d66e76e57b tail:0x140 end:0x140 dev:(efault) [ 1769.505054] ------------[ cut here ]------------ [ 1769.505723] kernel BUG at net/core/skbuff.c:111! (I am not sure, because I catch it on my branch)

EvgeniiMekhanik commented 9 months ago

[ 1080.197605] Start test: t_stress.test_stress.WrkStressMTU80.test_concurrent_connections [ 1080.320507] [tdb] Start Tempesta DB [ 1080.336831] [tempesta fw] Initializing Tempesta FW kernel module... [ 1080.409590] [tempesta fw] Configuration processing is completed. [ 1080.437556] [tdb] Opened table /opt/tempesta/db/filter0.tdb: size=16777216 rec_size=20 base=0000000028edbaa2 [ 1080.443528] [tdb] Opened table /opt/tempesta/db/sessions0.tdb: size=16777216 rec_size=312 base=000000000e8f334f [ 1080.449184] [tdb] Opened table /opt/tempesta/db/client0.tdb: size=16777216 rec_size=624 base=0000000084442b66 [ 1080.489070] [tempesta fw] Open listen socket on: 0.0.0.0 [ 1080.514064] [tempesta fw] Tempesta FW is ready [ 1087.061940] [tdb] Close table 'client0.tdb' [ 1087.065794] [tdb] Close table 'sessions0.tdb' [ 1087.109019] [tdb] Warning: Cannot sync mapping ffff95777a400000 of size 4096 pages [ 1087.110317] [tdb] Close table 'filter0.tdb' [ 1087.110939] [tempesta fw] modules are stopped [ 1087.112124] End test: t_stress.test_stress.WrkStressMTU80.test_concurrent_connections [ 1094.504680] Start test: http2_general.test_flow_control_window.TestFlowControl.test_flow_control_window_for_stream [ 1094.596234] [tempesta fw] Warning: Vhost default doesn't have certificate with matching SAN/CN. [ 1094.596234] Maybe that's fine, but it's worth checking the [ 1094.596234] config - if there is no relations between the [ 1094.596234] names, then host name confusion attack is possible. [ 1094.624782] [tempesta fw] Configuration processing is completed. [ 1094.656149] [tdb] Opened table /opt/tempesta/db/filter0.tdb: size=16777216 rec_size=20 base=0000000028edbaa2 [ 1094.743490] [tdb] Opened table /opt/tempesta/db/cache0.tdb: size=268435456 rec_size=0 base=00000000f28e6d2f [ 1094.746310] [tdb] Opened table /opt/tempesta/db/sessions0.tdb: size=16777216 rec_size=312 base=00000000c0b6aa0b [ 1094.749145] [tdb] Opened table /opt/tempesta/db/client0.tdb: size=16777216 rec_size=624 base=00000000d95b7053 [ 1094.768261] [tempesta fw] Open listen socket on: 0.0.0.0:443 [ 1094.784121] [tempesta fw] Tempesta FW is ready [ 1094.868309] ------------[ cut here ]------------ [ 1094.868902] kernel BUG at /home/evgeny/workdir/tempesta/fw/client.c:92! [ 1094.869688] invalid opcode: 0000 [#1] SMP NOPTI [ 1094.870229] CPU: 2 PID: 7441 Comm: python3 Tainted: G OE 5.10.35+ #275 [ 1094.870980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 1094.871490] RIP: 0010:tfw_client_put+0xa2/0xc0 [tempesta_fw] [ 1094.871830] Code: e8 53 37 26 d8 8b 05 ad 3a 0e 00 48 03 45 d8 4c 89 e7 48 89 83 58 01 00 00 e8 2e 4b 22 d8 66 90 65 48 ff 0d 78 75 31 3f eb a4 <0f> 0b 48 c7 c7 40 a3 dd c0 e8 5d 14 e2 d8 e8 bb f9 e6 d8 66 66 2e [ 1094.872946] RSP: 0018:ffffbb680012cb88 EFLAGS: 00010287 [ 1094.873259] RAX: ffff957778411018 RBX: ffff957768411018 RCX: 0000000000000004 [ 1094.873684] RDX: 0000000000000001 RSI: ffffffffc0da3b6d RDI: ffff957768411178 [ 1094.874108] RBP: ffffbb680012cbb0 R08: ffff9577490b9d00 R09: 0000000000000000 [ 1094.874556] R10: 0000000000000001 R11: 000000000014e600 R12: ffff957768411178 [ 1094.874980] R13: ffff95776841104c R14: ffff95778d23f720 R15: 0000000000000000 [ 1094.875405] FS: 00007f469b6fe000(0000) GS:ffff957877c80000(0000) knlGS:0000000000000000 [ 1094.875885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1094.876236] CR2: 00007f4696e30024 CR3: 0000000150a84000 CR4: 0000000000750ee0 [ 1094.876680] PKRU: 55555554 [ 1094.876846] Call Trace: [ 1094.876998] [ 1094.877131] tfw_classify_conn_close+0x5e/0x90 [tempesta_fw] [ 1094.877475] ss_conn_drop_guard_exit+0x3e/0x70 [tempesta_fw] [ 1094.877816] tcp_done+0xc9/0x140 [ 1094.878013] tcp_rcv_state_process+0xd0c/0x11c0 [ 1094.878287] ? sk_filter_trim_cap+0xde/0x240 [ 1094.878558] ? tcp_v4_inbound_md5_hash+0x57/0x1b0 [ 1094.878841] tcp_v4_do_rcv+0xc4/0x200 [ 1094.879063] tcp_v4_rcv+0xcd0/0xe20 [ 1094.879276] ip_protocol_deliver_rcu+0x44/0x230 [ 1094.879548] ip_local_deliver_finish+0x4e/0x70 [ 1094.879816] ip_local_deliver+0xf8/0x110 [ 1094.880055] ? ip_protocol_deliver_rcu+0x230/0x230 [ 1094.880345] ip_rcv_finish+0x87/0xa0 [ 1094.880562] ip_rcv+0xce/0xe0 [ 1094.880744] ? ip_rcv_finish_core.constprop.0+0x470/0x470 [ 1094.881068] netif_receive_skb_one_core+0x86/0xa0 [ 1094.881361] netif_receive_skb+0x18/0x60 [ 1094.881609] process_backlog+0x9e/0x170 [ 1094.881841] net_rx_action+0x13b/0x430 [ 1094.882070] __do_softirq+0xe3/0x340 [ 1094.882287] asm_call_irq_on_stack+0xf/0x20

EvgeniiMekhanik commented 9 months ago

This bug was reproduced after https://github.com/tempesta-tech/tempesta/pull/2039 so it is another BUG

EvgeniiMekhanik commented 9 months ago

Bug with clint.c will be fixed by https://github.com/tempesta-tech/tempesta/pull/2021/commits/3fab38ac3168d39be9a7ecd42a93d7b06af6c6fc. The problem was that when we obtain client after test was killed by OOM killer, we deal with already exist DB. When we obtain client from already exist old DB, this structure is not initialized correctly. Currently we just simply remove DB, before start tempesta, later it should be fixed in other way

krizhanovsky commented 9 months ago

Done in https://github.com/tempesta-tech/tempesta/pull/2041