tempesta-tech / tempesta

All-in-one solution for high performance web content delivery and advanced protection against DDoS and web attacks
https://tempesta-tech.com/
GNU General Public License v2.0
621 stars 103 forks source link

Large responses support #1714

Open b3b opened 2 years ago

b3b commented 2 years ago

Motivation

Currently, it is not possible to serve big responses (files) from backend, because of memory allocation error.

DMESG WARNING mm/page_alloc.c:4935
[53957.157446] [tdb] Start Tempesta DB
[53957.245089] [tempesta fw] Initializing Tempesta FW kernel module...
[53957.295412] [tempesta fw] Configuration processing is completed.
[53957.354527] [tdb] Opened table /opt/tempesta/db/filter0.tdb: size=16777216 rec_size=20 base=000000007718774f
[53957.396421] [tdb] Opened table /opt/tempesta/db/sessions0.tdb: size=16777216 rec_size=312 base=000000003e516204
[53957.431145] [tdb] Opened table /opt/tempesta/db/client0.tdb: size=16777216 rec_size=624 base=0000000080641d15
[53957.455301] [tempesta fw] Open listen socket on: 0.0.0.0
[53957.471289] [tempesta fw] Tempesta FW is ready
[53967.002299] ------------[ cut here ]------------
[53967.003668] WARNING: CPU: 5 PID: 6413 at mm/page_alloc.c:4935 __alloc_pages_nodemask+0x297/0x330
[53967.004935] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat br_netfilter bridge stp llc overlay dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr intel_rapl_common isst_if_common nfit rapl snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_hda_core snd_hwdep joydev input_leds snd_pcm snd_timer serio_raw snd soundcore mac_hid qemu_fw_cfg nf_log_ipv6 ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt tcp_bbr nf_log_ipv4 sch_fq nf_log_common ipt_REJECT nf_reject_ipv4 xt_LOG xt_limit xt_addrtype xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6table_filter ip6_tables iptable_filter bpfilter ipmi_msghandler msr ramoops reed_solomon efi_pstore virtio_rng ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
[53967.004991]  raid6_pq libcrc32c raid1 raid0 multipath linear mpt3sas raid_class scsi_transport_sas megaraid_sas hid_generic usbhid hid bochs_drm drm_vram_helper drm_ttm_helper crct10dif_pclmul crc32_pclmul ttm ghash_clmulni_intel drm_kms_helper syscopyarea sysfillrect aesni_intel crypto_simd sysimgblt fb_sys_fops cec cryptd glue_helper drm ahci libahci psmouse virtio_net net_failover i2c_i801 lpc_ich i2c_smbus failover virtio_blk [last unloaded: tempesta_lib]
[53967.017311] CPU: 5 PID: 6413 Comm: nginx Kdump: loaded Tainted: G           OE     5.10.35+ #1
[53967.018547] Hardware name: Vultr VC2, BIOS  
[53967.019778] RIP: 0010:__alloc_pages_nodemask+0x297/0x330
[53967.020945] Code: 08 44 89 f9 44 89 ea 4c 89 f6 e8 54 93 fc ff e9 4c ff ff ff e8 1a 14 fc ff 48 89 c7 e9 b8 fe ff ff 41 81 e4 00 20 00 00 75 a7 <0f> 0b eb a3 31 c0 e9 1b fe ff ff 65 48 8b 04 25 c0 7b 01 00 8b 40
[53967.023298] RSP: 0018:ffff9e66c01b8898 EFLAGS: 00010246
[53967.024451] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[53967.025576] RDX: 0000000000000000 RSI: 000000000000000b RDI: 0000000000000a20
[53967.035522] RBP: ffff9e66c01b88f0 R08: 0000000000000001 R09: 0000000000400010
[53967.036620] R10: 0000000000000000 R11: ffff9e66c01b8abc R12: 0000000000000000
[53967.037760] R13: ffffffffba7cab40 R14: 000000000000000b R15: 0000000000000000
[53967.038841] FS:  00007f9863ef1b80(0000) GS:ffff8a646fd40000(0000) knlGS:0000000000000000
[53967.039909] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[53967.040953] CR2: 000055f77348a620 CR3: 000000012cfa4004 CR4: 00000000005706e0
[53967.041996] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[53967.042976] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[53967.043964] PKRU: 55555554
[53967.044900] Call Trace:
[53967.045824]  
[53967.046789]  alloc_pages_current+0x87/0xe0
[53967.047744]  __get_free_pages+0x11/0x40
[53967.048642]  tfw_pool_alloc_pages+0x47/0x50 [tempesta_fw]
[53967.049567]  __tfw_pool_alloc_page+0x3d/0x90 [tempesta_fw]
[53967.050419]  __tfw_pool_realloc+0xd4/0xe0 [tempesta_fw]
[53967.051296]  __str_grow_tree+0x4a/0x140 [tempesta_fw]
[53967.052109]  tfw_str_add_compound+0x1b/0x20 [tempesta_fw]
[53967.052888]  __tfw_http_msg_add_str_data+0x3d/0xe0 [tempesta_fw]
[53967.053677]  tfw_http_parse_resp+0x966/0xfdc0 [tempesta_fw]
[53967.054420]  ss_skb_process+0xf6/0x140 [tempesta_fw]
[53967.055172]  ? __resp_parse_cache_control+0x1300/0x1300 [tempesta_fw]
[53967.055937]  tfw_http_msg_process_generic+0xcd/0x670 [tempesta_fw]
[53967.056678]  ? pg_skb_alloc+0x43b/0x480
[53967.057422]  ? __alloc_skb+0x44/0x200
[53967.058139]  tfw_http_msg_process+0x38/0x50 [tempesta_fw]
[53967.058846]  tfw_connection_recv+0x59/0xa0 [tempesta_fw]
[53967.059578]  ss_tcp_process_data+0x1e7/0x3f0 [tempesta_fw]
[53967.060280]  ss_tcp_data_ready+0x44/0xb0 [tempesta_fw]
[53967.060986]  tcp_data_ready+0x2b/0xd0
[53967.061721]  tcp_rcv_established+0x5a7/0x670
[53967.062403]  ? sk_filter_trim_cap+0xde/0x240
[53967.063063]  tcp_v4_do_rcv+0x140/0x200
[53967.063732]  tcp_v4_rcv+0xcfd/0xe10
[53967.064342]  ip_protocol_deliver_rcu+0x30/0x1b0
[53967.064938]  ip_local_deliver_finish+0x48/0x60
[53967.065504]  ip_local_deliver+0xfa/0x110
[53967.066110]  ? ip_protocol_deliver_rcu+0x1b0/0x1b0
[53967.066653]  ip_rcv_finish+0x87/0xa0
[53967.067191]  ip_rcv+0xcc/0xe0
[53967.067723]  ? ip_rcv_finish_core.isra.0+0x420/0x420
[53967.068205]  __netif_receive_skb_one_core+0x88/0xa0
[53967.068660]  __netif_receive_skb+0x18/0x60
[53967.069095]  process_backlog+0xa9/0x160
[53967.069528]  net_rx_action+0x13e/0x390
[53967.069963]  ? clockevents_program_event+0x8f/0xe0
[53967.070374]  __do_softirq+0xd9/0x291
[53967.070773]  asm_call_irq_on_stack+0x12/0x20
[53967.071159]  
[53967.071584]  do_softirq_own_stack+0x3d/0x50
[53967.071995]  do_softirq.part.0+0x46/0x50
[53967.072380]  __local_bh_enable_ip+0x50/0x60
[53967.072771]  ip_finish_output2+0x1ab/0x590
[53967.073157]  ? __cgroup_bpf_run_filter_skb+0x3c3/0x3d0
[53967.073570]  __ip_finish_output+0xd8/0x220
[53967.073950]  ip_finish_output+0x2d/0xb0
[53967.074315]  ip_output+0x7a/0x100
[53967.074690]  ? __ip_finish_output+0x220/0x220
[53967.075052]  ip_local_out+0x3d/0x50
[53967.075484]  __ip_queue_xmit+0x17a/0x470
[53967.075865]  ip_queue_xmit+0x15/0x20
[53967.076231]  __tcp_transmit_skb+0xa20/0xbe0
[53967.076598]  tcp_write_xmit+0x453/0x1350
[53967.076970]  tcp_push_one+0x3c/0x40
[53967.077365]  do_tcp_sendpages+0x34d/0x640
[53967.077741]  tcp_sendpage_locked+0x43/0x60
[53967.078117]  tcp_sendpage+0x3c/0x60
[53967.078478]  inet_sendpage+0x56/0xa0
[53967.078843]  kernel_sendpage+0x57/0xc0
[53967.079196]  ? atime_needs_update+0x73/0xd0
[53967.079572]  sock_sendpage+0x24/0x30
[53967.079961]  pipe_to_sendpage+0x61/0xa0
[53967.080297]  __splice_from_pipe+0xa9/0x1c0
[53967.080634]  ? propagate_umount+0x4f0/0x4f0
[53967.080962]  ? propagate_umount+0x4f0/0x4f0
[53967.081282]  splice_from_pipe+0x5f/0x90
[53967.081600]  generic_splice_sendpage+0x15/0x20
[53967.081924]  direct_splice_actor+0x30/0x50
[53967.082239]  splice_direct_to_actor+0xfa/0x250
[53967.082578]  ? wait_for_space+0x90/0x90
[53967.082906]  do_splice_direct+0x89/0xd0
[53967.083277]  do_sendfile+0x1b2/0x470
[53967.083626]  __x64_sys_sendfile64+0x62/0xc0
[53967.083959]  do_syscall_64+0x38/0x90
[53967.084330]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[53967.084680] RIP: 0033:0x7f9864005fae
[53967.085001] Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 bd fd ff ff 0f 1f 44 00 00 31 c0 c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d b2 8e 0d 00 f7 d8 64 89 01 48
[53967.085771] RSP: 002b:00007ffcb2178a28 EFLAGS: 00000246 ORIG_RAX: 0000000000000028
[53967.086145] RAX: ffffffffffffffda RBX: 00007ffcb2178a78 RCX: 00007f9864005fae
[53967.086522] RDX: 00007ffcb2178a78 RSI: 0000000000000018 RDI: 0000000000000017
[53967.086919] RBP: 000055f774e44118 R08: 0000000040000000 R09: 0000000040000000
[53967.087344] R10: 0000000040000000 R11: 0000000000000246 R12: 000055f774e6c7c0
[53967.087761] R13: 0000000040000000 R14: 000055f774e43f80 R15: 00007ffcb2178a80
[53967.088162] ---[ end trace 774c838d6c8445cd ]---
[53967.088549] [tempesta fw] Warning: Cannot grow HTTP data string
[53967.092199] [tempesta fw] Warning: response dropped: processing error: 127.0.0.1
[53967.092749] [tempesta fw] Warning: Paired request missing, HTTP Response Splitting attack?

Related issue: https://github.com/tempesta-tech/tempesta/issues/498

Testing

To reproduce, backend on port 8000 should return a large response (tested on 1GB).

tempesta.cfg

listen 80 proto=http;
server 127.0.0.1:8000;
cache 0;

curl -v http://127.0.0.1/test


*   Trying 127.0.0.1:80...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET /test HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< date: Tue, 11 Oct 2022 13:51:41 GMT
< content-length: 0
< server: Tempesta FW/pre-0.7.0
< 
* Connection #0 to host 127.0.0.1 left intact
RomanBelozerov commented 2 years ago

I received this bug for h2 request with 500+ MB in body. Test to reproduce.

dmesg ``` [ 283.670967] ------------[ cut here ]------------ [ 283.671949] WARNING: CPU: 5 PID: 3051 at mm/page_alloc.c:4935 __alloc_pages_nodemask+0x268/0x300 [ 283.673730] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 rfcomm bnep vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock nls_iso8859_1 binfmt_misc intel_rapl_msr intel_rapl_common crct10dif_pclmul ghash_clmulni_intel aesni_intel snd_ens1371 crypto_simd snd_ac97_codec cryptd gameport glue_helper ac97_bus snd_pcm btusb btrtl snd_seq_midi snd_seq_midi_event btbcm btintel snd_rawmidi bluetooth snd_seq vmw_balloon snd_seq_device joydev snd_timer input_leds serio_raw snd ecdh_generic ecc soundcore vmw_vmci sch_fq_codel vmwgfx mac_hid ttm drm_kms_helper cec rc_core fb_sys_fops syscopyarea sysfillrect sysimgblt ipmi_devintf ipmi_msghandler drm msr parport_pc ppdev lp parport ramoops pstore_blk reed_solomon efi_pstore pstore_zone ip_tables x_tables autofs4 hid_generic usbhid hid mptspi psmouse crc32_pclmul e1000 mptscsih ahci mptbase i2c_piix4 libahci scsi_transport_spi pata_acpi [ 283.673786] [last unloaded: tempesta_lib] [ 283.691803] CPU: 5 PID: 3051 Comm: curl Kdump: loaded Tainted: G OE 5.10.35+ #1 [ 283.693529] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020 [ 283.695779] RIP: 0010:__alloc_pages_nodemask+0x268/0x300 [ 283.696892] Code: 08 44 89 f9 44 89 ea 4c 89 f6 e8 23 95 fc ff e9 4c ff ff ff e8 c9 15 fc ff 48 89 c7 e9 b8 fe ff ff 41 81 e4 00 20 00 00 75 a7 <0f> 0b eb a3 31 c0 e9 4a fe ff ff 65 48 8b 04 25 c0 7b 01 00 8b 40 [ 283.700779] RSP: 0018:ffffb856405b0758 EFLAGS: 00010246 [ 283.701876] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 [ 283.703435] RDX: 0000000000000000 RSI: 000000000000000b RDI: 0000000000000a20 [ 283.704900] RBP: ffffb856405b07b0 R08: 0000000000000001 R09: 0000000000400010 [ 283.706438] R10: ffff8d76662a20f0 R11: 0000000000000009 R12: 0000000000000000 [ 283.707860] R13: ffffffff8c9d1c40 R14: 000000000000000b R15: 0000000000000000 [ 283.709308] FS: 00007f0ba9486b80(0000) GS:ffff8d7775f40000(0000) knlGS:0000000000000000 [ 283.711057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 283.712236] CR2: 00007f0ba9a2a520 CR3: 000000000115a006 CR4: 0000000000370ee0 [ 283.713665] Call Trace: [ 283.714294] [ 283.714746] alloc_pages_current+0x87/0xe0 [ 283.715584] __get_free_pages+0x11/0x40 [ 283.716390] tfw_pool_alloc_pages+0x47/0x50 [tempesta_fw] [ 283.717825] __tfw_pool_alloc_page+0x3d/0x90 [tempesta_fw] [ 283.719387] __tfw_pool_realloc+0xd4/0xe0 [tempesta_fw] [ 283.721090] __str_grow_tree+0x4a/0x140 [tempesta_fw] [ 283.722279] tfw_str_add_compound+0x1b/0x20 [tempesta_fw] [ 283.723441] __tfw_http_msg_add_str_data+0x3d/0xe0 [tempesta_fw] [ 283.724747] tfw_h2_parse_req+0xf0/0x1c0 [tempesta_fw] [ 283.725852] ss_skb_process+0xf6/0x140 [tempesta_fw] [ 283.727057] ? tfw_http_parse_req+0xe1a0/0xe1a0 [tempesta_fw] [ 283.728319] ? tfw_http_parse_req+0xe1a0/0xe1a0 [tempesta_fw] [ 283.729552] tfw_http_req_process+0x7f/0x960 [tempesta_fw] [ 283.730854] ? tfw_h2_frame_recv+0x243/0x1f80 [tempesta_fw] [ 283.732065] tfw_http_msg_process_generic+0x197/0x670 [tempesta_fw] [ 283.733409] ? ss_skb_process+0xf6/0x140 [tempesta_fw] [ 283.734639] ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw] [ 283.735990] tfw_h2_frame_process+0x308/0x440 [tempesta_fw] [ 283.737193] tfw_http_msg_process+0x42/0x50 [tempesta_fw] [ 283.738468] tfw_connection_recv+0x59/0xa0 [tempesta_fw] [ 283.739659] tfw_tls_connection_recv+0x19d/0x380 [tempesta_fw] [ 283.740918] ss_tcp_process_data+0x1e7/0x3f0 [tempesta_fw] [ 283.742090] ss_tcp_data_ready+0x44/0xb0 [tempesta_fw] [ 283.743438] tcp_data_ready+0x2b/0xd0 [ 283.744452] tcp_rcv_established+0x5a7/0x670 [ 283.745453] ? sk_filter_trim_cap+0xde/0x240 [ 283.746536] tcp_v4_do_rcv+0x140/0x200 [ 283.747401] tcp_v4_rcv+0xcfd/0xe10 [ 283.748176] ip_protocol_deliver_rcu+0x30/0x1b0 [ 283.749187] ip_local_deliver_finish+0x48/0x60 [ 283.750165] ip_local_deliver+0x72/0x110 [ 283.751152] ? tfw_ipv4_nf_hook+0xb9/0x130 [tempesta_fw] [ 283.752340] ip_rcv_finish+0x87/0xa0 [ 283.753125] ip_rcv+0xcc/0xe0 [ 283.753909] ? ip_rcv_finish_core.isra.0+0x420/0x420 [ 283.755009] __netif_receive_skb_one_core+0x88/0xa0 [ 283.756081] __netif_receive_skb+0x18/0x60 [ 283.756985] process_backlog+0xa9/0x160 [ 283.757946] net_rx_action+0x13e/0x390 [ 283.758621] ? rcu_core+0xfa/0x500 [ 283.759239] __do_softirq+0xd9/0x291 [ 283.759870] asm_call_irq_on_stack+0x12/0x20 [ 283.760800] [ 283.761270] do_softirq_own_stack+0x3d/0x50 [ 283.762313] do_softirq.part.0+0x46/0x50 [ 283.763169] __local_bh_enable_ip+0x50/0x60 [ 283.764111] ip_finish_output2+0x1ab/0x590 [ 283.764947] ? __cgroup_bpf_run_filter_skb+0x3c3/0x3d0 [ 283.766019] __ip_finish_output+0xd8/0x220 [ 283.766734] ? __wake_up_common+0x7e/0x140 [ 283.767666] ip_finish_output+0x2d/0xb0 [ 283.768501] ip_output+0x7a/0x100 [ 283.769457] ? set_pageblock_migratetype+0x39/0x40 [ 283.771861] ip_local_out+0x3d/0x50 [ 283.773168] __ip_queue_xmit+0x17a/0x470 [ 283.774533] ? get_page_from_freelist+0xed8/0x1100 [ 283.776049] ip_queue_xmit+0x15/0x20 [ 283.776893] __tcp_transmit_skb+0xa94/0xc90 [ 283.778065] tcp_write_xmit+0x453/0x1350 [ 283.779036] __tcp_push_pending_frames+0x37/0x100 [ 283.780130] tcp_push+0xfc/0x100 [ 283.780839] tcp_sendmsg_locked+0xd36/0xe70 [ 283.781795] tcp_sendmsg+0x2d/0x50 [ 283.782430] inet_sendmsg+0x43/0x70 [ 283.783174] sock_sendmsg+0x5e/0x70 [ 283.783935] sock_write_iter+0x93/0xf0 [ 283.784759] new_sync_write+0x192/0x1b0 [ 283.785732] vfs_write+0x1ca/0x280 [ 283.786514] ksys_write+0xb1/0xe0 [ 283.787245] __x64_sys_write+0x1a/0x20 [ 283.788066] do_syscall_64+0x38/0x90 [ 283.788844] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 283.790076] RIP: 0033:0x7f0ba9a272f7 [ 283.790974] Code: 75 05 48 83 c4 58 c3 e8 f7 33 ff ff 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 283.795149] RSP: 002b:00007fff282ba368 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 283.796847] RAX: ffffffffffffffda RBX: 0000000000000026 RCX: 00007f0ba9a272f7 [ 283.798508] RDX: 0000000000000026 RSI: 000055d9f9228c83 RDI: 0000000000000006 [ 283.800067] RBP: 000055d9f92009c0 R08: 0000000000000001 R09: 0000000000000008 [ 283.801688] R10: 000055d9f9228c85 R11: 0000000000000246 R12: 000055d9f9228c83 [ 283.803244] R13: 0000000000000026 R14: 000055d9f91fed10 R15: 0000000000000000 [ 283.804781] ---[ end trace 9dcd1d8884faa490 ]--- [ 283.805907] [tempesta fw] Warning: Cannot grow HTTP data string [ 283.807127] [tempesta fw] Warning: failed to parse request: 127.0.0.1 [ 283.808552] ------------[ cut here ]------------ [ 283.809602] WARNING: CPU: 5 PID: 3051 at /root/tempesta/fw/http_stream.c:89 tfw_h2_stream_fsm+0x12d/0x2b0 [tempesta_fw] [ 283.811886] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 rfcomm bnep vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock nls_iso8859_1 binfmt_misc intel_rapl_msr intel_rapl_common crct10dif_pclmul ghash_clmulni_intel aesni_intel snd_ens1371 crypto_simd snd_ac97_codec cryptd gameport glue_helper ac97_bus snd_pcm btusb btrtl snd_seq_midi snd_seq_midi_event btbcm btintel snd_rawmidi bluetooth snd_seq vmw_balloon snd_seq_device joydev snd_timer input_leds serio_raw snd ecdh_generic ecc soundcore vmw_vmci sch_fq_codel vmwgfx mac_hid ttm drm_kms_helper cec rc_core fb_sys_fops syscopyarea sysfillrect sysimgblt ipmi_devintf ipmi_msghandler drm msr parport_pc ppdev lp parport ramoops pstore_blk reed_solomon efi_pstore pstore_zone ip_tables x_tables autofs4 hid_generic usbhid hid mptspi psmouse crc32_pclmul e1000 mptscsih ahci mptbase i2c_piix4 libahci scsi_transport_spi pata_acpi [ 283.811940] [last unloaded: tempesta_lib] [ 283.828912] CPU: 5 PID: 3051 Comm: curl Kdump: loaded Tainted: G W OE 5.10.35+ #1 [ 283.830414] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020 [ 283.832161] RIP: 0010:tfw_h2_stream_fsm+0x12d/0x2b0 [tempesta_fw] [ 283.833149] Code: 89 a0 01 00 00 0f 0b 83 f8 06 74 6b 83 f8 07 75 f4 45 31 c0 41 80 fd 02 0f 84 56 ff ff ff 41 80 fd 09 0f 85 46 ff ff ff eb 8c <0f> 0b 41 b8 03 00 00 00 e9 3d ff ff ff 41 b8 03 00 00 00 e9 3c ff [ 283.836301] RSP: 0018:ffffb856405b0530 EFLAGS: 00010293 [ 283.837153] RAX: 0000000000000002 RBX: ffff8d76e04b82c0 RCX: 0000000000000001 [ 283.838317] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d76e04b82f0 [ 283.839674] RBP: ffffb856405b0560 R08: ffffb856405b057c R09: 0000000000000000 [ 283.840858] R10: 00000000ffffffff R11: a3d70a3d70a3d70b R12: ffff8d76e04b82f0 [ 283.842062] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001 [ 283.843418] FS: 00007f0ba9486b80(0000) GS:ffff8d7775f40000(0000) knlGS:0000000000000000 [ 283.844765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 283.845757] CR2: 00007f0ba9a2a520 CR3: 000000000115a006 CR4: 0000000000370ee0 [ 283.847385] Call Trace: [ 283.847916] [ 283.848370] tfw_h2_stream_id_close+0xe7/0x110 [tempesta_fw] [ 283.849429] tfw_h2_prep_resp+0x95/0x310 [tempesta_fw] [ 283.850305] ? alloc_pages_current+0x87/0xe0 [ 283.851187] tfw_h2_send_resp+0x5c/0xb0 [tempesta_fw] [ 283.852045] tfw_h2_send_err_resp+0x9a/0xc0 [tempesta_fw] [ 283.852926] tfw_http_cli_error_resp_and_log+0x3cd/0x440 [tempesta_fw] [ 283.854200] ? tfw_http_parse_req+0xe1a0/0xe1a0 [tempesta_fw] [ 283.855181] tfw_http_req_process+0xf6/0x960 [tempesta_fw] [ 283.856064] ? tfw_h2_frame_recv+0x243/0x1f80 [tempesta_fw] [ 283.856963] tfw_http_msg_process_generic+0x197/0x670 [tempesta_fw] [ 283.857998] ? ss_skb_process+0xf6/0x140 [tempesta_fw] [ 283.858920] ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw] [ 283.860019] tfw_h2_frame_process+0x308/0x440 [tempesta_fw] [ 283.860916] tfw_http_msg_process+0x42/0x50 [tempesta_fw] [ 283.861806] tfw_connection_recv+0x59/0xa0 [tempesta_fw] [ 283.862787] tfw_tls_connection_recv+0x19d/0x380 [tempesta_fw] [ 283.863808] ss_tcp_process_data+0x1e7/0x3f0 [tempesta_fw] [ 283.864697] ss_tcp_data_ready+0x44/0xb0 [tempesta_fw] [ 283.865730] tcp_data_ready+0x2b/0xd0 [ 283.866423] tcp_rcv_established+0x5a7/0x670 [ 283.867112] ? sk_filter_trim_cap+0xde/0x240 [ 283.867802] tcp_v4_do_rcv+0x140/0x200 [ 283.868416] tcp_v4_rcv+0xcfd/0xe10 [ 283.868984] ip_protocol_deliver_rcu+0x30/0x1b0 [ 283.869740] ip_local_deliver_finish+0x48/0x60 [ 283.870586] ip_local_deliver+0x72/0x110 [ 283.871268] ? tfw_ipv4_nf_hook+0xb9/0x130 [tempesta_fw] [ 283.872150] ip_rcv_finish+0x87/0xa0 [ 283.872744] ip_rcv+0xcc/0xe0 [ 283.873248] ? ip_rcv_finish_core.isra.0+0x420/0x420 [ 283.874229] __netif_receive_skb_one_core+0x88/0xa0 [ 283.875037] __netif_receive_skb+0x18/0x60 [ 283.875723] process_backlog+0xa9/0x160 [ 283.876395] net_rx_action+0x13e/0x390 [ 283.877056] ? rcu_core+0xfa/0x500 [ 283.877751] __do_softirq+0xd9/0x291 [ 283.878352] asm_call_irq_on_stack+0x12/0x20 [ 283.879041] [ 283.879391] do_softirq_own_stack+0x3d/0x50 [ 283.880064] do_softirq.part.0+0x46/0x50 [ 283.880704] __local_bh_enable_ip+0x50/0x60 [ 283.881385] ip_finish_output2+0x1ab/0x590 [ 283.882183] ? __cgroup_bpf_run_filter_skb+0x3c3/0x3d0 [ 283.883126] __ip_finish_output+0xd8/0x220 [ 283.883794] ? __wake_up_common+0x7e/0x140 [ 283.884458] ip_finish_output+0x2d/0xb0 [ 283.885080] ip_output+0x7a/0x100 [ 283.885620] ? set_pageblock_migratetype+0x39/0x40 [ 283.886566] ip_local_out+0x3d/0x50 [ 283.887167] __ip_queue_xmit+0x17a/0x470 [ 283.887837] ? get_page_from_freelist+0xed8/0x1100 [ 283.888668] ip_queue_xmit+0x15/0x20 [ 283.889278] __tcp_transmit_skb+0xa94/0xc90 [ 283.890124] tcp_write_xmit+0x453/0x1350 [ 283.890787] __tcp_push_pending_frames+0x37/0x100 [ 283.891543] tcp_push+0xfc/0x100 [ 283.892068] tcp_sendmsg_locked+0xd36/0xe70 [ 283.892777] tcp_sendmsg+0x2d/0x50 [ 283.893334] inet_sendmsg+0x43/0x70 [ 283.894049] sock_sendmsg+0x5e/0x70 [ 283.894637] sock_write_iter+0x93/0xf0 [ 283.895244] new_sync_write+0x192/0x1b0 [ 283.895869] vfs_write+0x1ca/0x280 [ 283.896423] ksys_write+0xb1/0xe0 [ 283.896961] __x64_sys_write+0x1a/0x20 [ 283.897744] do_syscall_64+0x38/0x90 [ 283.898454] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 283.899323] RIP: 0033:0x7f0ba9a272f7 [ 283.899909] Code: 75 05 48 83 c4 58 c3 e8 f7 33 ff ff 0f 1f 80 00 00 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [ 283.903020] RSP: 002b:00007fff282ba368 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 283.904303] RAX: ffffffffffffffda RBX: 0000000000000026 RCX: 00007f0ba9a272f7 [ 283.905483] RDX: 0000000000000026 RSI: 000055d9f9228c83 RDI: 0000000000000006 [ 283.907016] RBP: 000055d9f92009c0 R08: 0000000000000001 R09: 0000000000000008 [ 283.908470] R10: 000055d9f9228c85 R11: 0000000000000246 R12: 000055d9f9228c83 [ 283.910047] R13: 0000000000000026 R14: 000055d9f91fed10 R15: 0000000000000000 [ 283.911324] ---[ end trace 9dcd1d8884faa491 ]--- ```

tls.test_tls_integrity.ProxyH2 with TCP segmentation 1 byte. Tempesta - f6d4355dfb9040473fa0c5e84891d207441babf6

[  148.472087] ------------[ cut here ]------------
[  148.472944] WARNING: CPU: 2 PID: 2447 at mm/page_alloc.c:4935 __alloc_pages_nodemask+0x291/0x310
[  148.475965] Modules linked in: tempesta_fw(OE) tempesta_db(OE) sha256_ssse3(E) sha512_ssse3(E) sha512_generic(E) tempesta_tls(OE) tempesta_lib(OE) uinput(E) xt_conntrack(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) nf_conntrack_netlink(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) xfrm_user(E) xfrm_algo(E) nft_counter(E) xt_addrtype(E) nft_compat(E) nf_tables(E) libcrc32c(E) nfnetlink(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) hid_generic(E) usbhid(E) hid(E) rfkill(E) intel_rapl_msr(E) intel_rapl_common(E) intel_pmc_core(E) crc32_pclmul(E) ghash_clmulni_intel(E) sr_mod(E) snd_ens1371(E) cdrom(E) snd_ac97_codec(E) aesni_intel(E) libaes(E) ata_generic(E) crypto_simd(E) ac97_bus(E) cryptd(E) gameport(E) glue_helper(E) vsock_loopback(E) snd_rawmidi(E) vmw_vsock_virtio_transport_common(E) snd_seq_device(E) ehci_pci(E) uhci_hcd(E) snd_pcm(E) ehci_hcd(E) vmw_balloon(E) ata_piix(E) snd_timer(E) snd(E) usbcore(E) joydev(E) pcspkr(E) libata(E) e1000(E)
[  148.476005]  vmw_vsock_vmci_transport(E) soundcore(E) i2c_piix4(E) vsock(E) sg(E) vmw_vmci(E) button(E) ac(E) msr(E) parport_pc(E) ppdev(E) lp(E) parport(E) fuse(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc32c_generic(E) crc16(E) mbcache(E) jbd2(E) sd_mod(E) t10_pi(E) crc_t10dif(E) crct10dif_generic(E) vmwgfx(E) drm_kms_helper(E) cec(E) ttm(E) drm(E) mptspi(E) scsi_transport_spi(E) mptscsih(E) mptbase(E) scsi_mod(E) psmouse(E) crct10dif_pclmul(E) crct10dif_common(E) evdev(E) crc32c_intel(E) serio_raw(E)
[  148.499405] CPU: 2 PID: 2447 Comm: python3 Tainted: G           OE     5.10.35+ #16
[  148.500864] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[  148.502789] RIP: 0010:__alloc_pages_nodemask+0x291/0x310
[  148.503852] Code: 48 8b 04 25 c0 7b 01 00 48 05 48 0c 00 00 41 bd 01 00 00 00 48 89 44 24 08 e9 47 fe ff ff 81 e5 00 20 00 00 0f 85 60 ff ff ff <0f> 0b e9 59 ff ff ff 31 c0 e9 1b fe ff ff e8 4c 41 fc ff 48 89 c7
[  148.507173] RSP: 0018:ffffb85fc04eca50 EFLAGS: 00010246
[  148.508360] RAX: 0000000000000000 RBX: ffff9c2925286010 RCX: 0000000000000000
[  148.510285] RDX: 0000000000000000 RSI: 000000000000000b RDI: 0000000000040a20
[  148.512192] RBP: 0000000000000000 R08: 0000000000000001 R09: 00000000007ffff0
[  148.513804] R10: 0000000000000020 R11: 0000000000000001 R12: 0000000000400010
[  148.515284] R13: 000000000000000b R14: ffffb85fc04ecbe4 R15: ffff9c270db3e6c0
[  148.516598] FS:  00007fe46910d700(0000) GS:ffff9c295a080000(0000) knlGS:0000000000000000
[  148.518163] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  148.519440] CR2: 00007fd854646ccc CR3: 00000001901ee002 CR4: 0000000000370ee0
[  148.520741] Call Trace:
[  148.521257]  <IRQ>
[  148.521654]  __get_free_pages+0xd/0x30
[  148.522372]  __tfw_pool_realloc+0xd3/0x120 [tempesta_fw]
[  148.523425]  __str_grow_tree+0x45/0x140 [tempesta_fw]
[  148.524340]  __tfw_http_msg_add_str_data+0x38/0xd0 [tempesta_fw]
[  148.525435]  tfw_h2_parse_req+0x138/0x200 [tempesta_fw]
[  148.526374]  ss_skb_process+0xed/0x130 [tempesta_fw]
[  148.527303]  ? h2_set_hdr_if_mod_since+0x40/0x40 [tempesta_fw]
[  148.528431]  tfw_http_req_process+0x7c/0x870 [tempesta_fw]
[  148.529439]  ? ttls_decrypt+0x28e/0x510 [tempesta_tls]
[  148.530424]  ? ss_skb_process+0xed/0x130 [tempesta_fw]
[  148.531359]  ? tfw_h2_conn_terminate_close+0x30/0x30 [tempesta_fw]
[  148.532947]  tfw_h2_frame_process+0x26b/0x4b0 [tempesta_fw]
[  148.534036]  tfw_connection_recv+0x52/0xa0 [tempesta_fw]
[  148.535215]  tfw_tls_connection_recv+0x281/0x3a0 [tempesta_fw]
[  148.537516]  ss_tcp_process_data+0x1f1/0x400 [tempesta_fw]
[  148.538632]  ss_tcp_data_ready+0x3f/0xc0 [tempesta_fw]
[  148.539933]  tcp_rcv_established+0x5d8/0x680
[  148.540933]  tcp_v4_do_rcv+0x131/0x1f0
[  148.541662]  tcp_v4_rcv+0xc2f/0xd80
[  148.542292]  ip_protocol_deliver_rcu+0x2b/0x1b0
[  148.543269]  ip_local_deliver_finish+0x44/0x50
[  148.544085]  __netif_receive_skb_one_core+0x87/0xa0
[  148.544958]  process_backlog+0x96/0x160
[  148.545668]  net_rx_action+0x145/0x3e0
[  148.546344]  __do_softirq+0xcf/0x284
[  148.546987]  asm_call_irq_on_stack+0x12/0x20
[  148.547879]  </IRQ>
[  148.548328]  do_softirq_own_stack+0x37/0x40
[  148.549087]  do_softirq+0x5e/0x70
[  148.549701]  __local_bh_enable_ip+0x4b/0x50
[  148.550527]  ip_finish_output2+0x1ab/0x590
[  148.551284]  ? ipv4_link_failure+0x1b0/0x1b0
[  148.552145]  __ip_queue_xmit+0x180/0x410
[  148.552911]  __tcp_transmit_skb+0xa0e/0xbc0
[  148.553661]  tcp_write_xmit+0x401/0x1280
[  148.554376]  __tcp_push_pending_frames+0x32/0xf0
[  148.555203]  tcp_sendmsg_locked+0xa32/0xb50
[  148.556040]  tcp_sendmsg+0x28/0x40
[  148.556697]  sock_sendmsg+0x57/0x60
[  148.557353]  sock_write_iter+0x97/0x100
[  148.558063]  new_sync_write+0x199/0x1b0
[  148.558756]  vfs_write+0x1c2/0x260
[  148.559536]  ksys_write+0xa7/0xe0
[  148.560242]  ? exit_to_user_mode_prepare+0x6e/0x120
[  148.561277]  do_syscall_64+0x33/0x80
[  148.562051]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  148.563334] RIP: 0033:0x7fe46dcfffef
[  148.564021] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 29 fd ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 5c fd ff ff 48
[  148.567729] RSP: 002b:00007fe46910a730 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[  148.569246] RAX: ffffffffffffffda RBX: 000000000000001e RCX: 00007fe46dcfffef
[  148.570624] RDX: 000000000000001e RSI: 00007fe46406eb73 RDI: 0000000000000005
[  148.572108] RBP: 00007fe464003cd0 R08: 0000000000000000 R09: 00007fe46910a8b0
[  148.573656] R10: 00007fe46cfb1d50 R11: 0000000000000293 R12: 00007fe46406eb73
[  148.574995] R13: 000000000000001e R14: 00007fe464002808 R15: 00007fe464001f80
[  148.576443] ---[ end trace 4c9ceb94de080a36 ]---
[  148.577358] [tempesta fw] Warning: Cannot grow HTTP data string
[  148.578784] [tempesta fw] Warning: failed to parse request: 127.0.0.1
RomanBelozerov commented 1 year ago

TCP segmentation 1 for tests:

[Client] ip = 192.168.122.1 ipv6 = ::1 hostname = localhost ab = ab wrk = wrk h2load = h2load tls-perf = tls-perf workdir = /tmp/client unavaliable_timeout = 300

[Tempesta] ip = 192.168.122.116 ipv6 = ::1 hostname = 192.168.122.116 user = root port = 22 srcdir = /root/tempesta workdir = /tmp/host config = tempesta.conf tmp_config = tempesta_tmp.conf unavaliable_timeout = 300

[Server] ip = 192.168.122.1 ipv6 = ::1 hostname = localhost user = root port = 22 nginx = nginx workdir = /tmp/nginx resources = /var/www/html/ aliases_interface = virbr0 aliases_base_ip = 192.168.122.1 max_workers = 16 keepalive_timeout = 60 keepalive_requests = 100 unavaliable_timeout = 300