RomanBelozerov
commented
2 years ago I received this bug for h2 request with 500+ MB in body. Test to reproduce.
dmesg
``` [ 283.670967] ------------[ cut here ]------------ [ 283.671949] WARNING: CPU: 5 PID: 3051 at mm/page_alloc.c:4935 __alloc_pages_nodemask+0x268/0x300 [ 283.673730] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 rfcomm bnep vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock nls_iso8859_1 binfmt_misc intel_rapl_msr intel_rapl_common crct10dif_pclmul ghash_clmulni_intel aesni_intel snd_ens1371 crypto_simd snd_ac97_codec cryptd gameport glue_helper ac97_bus snd_pcm btusb btrtl snd_seq_midi snd_seq_midi_event btbcm btintel snd_rawmidi bluetooth snd_seq vmw_balloon snd_seq_device joydev snd_timer input_leds serio_raw snd ecdh_generic ecc soundcore vmw_vmci sch_fq_codel vmwgfx mac_hid ttm drm_kms_helper cec rc_core fb_sys_fops syscopyarea sysfillrect sysimgblt ipmi_devintf ipmi_msghandler drm msr parport_pc ppdev lp parport ramoops pstore_blk reed_solomon efi_pstore pstore_zone ip_tables x_tables autofs4 hid_generic usbhid hid mptspi psmouse crc32_pclmul e1000 mptscsih ahci mptbase i2c_piix4 libahci scsi_transport_spi pata_acpi [ 283.673786] [last unloaded: tempesta_lib] [ 283.691803] CPU: 5 PID: 3051 Comm: curl Kdump: loaded Tainted: G OE 5.10.35+ #1 [ 283.693529] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020 [ 283.695779] RIP: 0010:__alloc_pages_nodemask+0x268/0x300 [ 283.696892] Code: 08 44 89 f9 44 89 ea 4c 89 f6 e8 23 95 fc ff e9 4c ff ff ff e8 c9 15 fc ff 48 89 c7 e9 b8 fe ff ff 41 81 e4 00 20 00 00 75 a7 <0f> 0b eb a3 31 c0 e9 4a fe ff ff 65 48 8b 04 25 c0 7b 01 00 8b 40 [ 283.700779] RSP: 0018:ffffb856405b0758 EFLAGS: 00010246 [ 283.701876] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 [ 283.703435] RDX: 0000000000000000 RSI: 000000000000000b RDI: 0000000000000a20 [ 283.704900] RBP: ffffb856405b07b0 R08: 0000000000000001 R09: 0000000000400010 [ 283.706438] R10: ffff8d76662a20f0 R11: 0000000000000009 R12: 0000000000000000 [ 283.707860] R13: ffffffff8c9d1c40 R14: 000000000000000b R15: 0000000000000000 [ 283.709308] FS: 00007f0ba9486b80(0000) GS:ffff8d7775f40000(0000) knlGS:0000000000000000 [ 283.711057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 283.712236] CR2: 00007f0ba9a2a520 CR3: 000000000115a006 CR4: 0000000000370ee0 [ 283.713665] Call Trace: [ 283.714294]tls.test_tls_integrity.ProxyH2 with TCP segmentation 1 byte. Tempesta - f6d4355dfb9040473fa0c5e84891d207441babf6
[ 148.472087] ------------[ cut here ]------------
[ 148.472944] WARNING: CPU: 2 PID: 2447 at mm/page_alloc.c:4935 __alloc_pages_nodemask+0x291/0x310
[ 148.475965] Modules linked in: tempesta_fw(OE) tempesta_db(OE) sha256_ssse3(E) sha512_ssse3(E) sha512_generic(E) tempesta_tls(OE) tempesta_lib(OE) uinput(E) xt_conntrack(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_nat(E) nf_conntrack_netlink(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) xfrm_user(E) xfrm_algo(E) nft_counter(E) xt_addrtype(E) nft_compat(E) nf_tables(E) libcrc32c(E) nfnetlink(E) br_netfilter(E) bridge(E) stp(E) llc(E) overlay(E) hid_generic(E) usbhid(E) hid(E) rfkill(E) intel_rapl_msr(E) intel_rapl_common(E) intel_pmc_core(E) crc32_pclmul(E) ghash_clmulni_intel(E) sr_mod(E) snd_ens1371(E) cdrom(E) snd_ac97_codec(E) aesni_intel(E) libaes(E) ata_generic(E) crypto_simd(E) ac97_bus(E) cryptd(E) gameport(E) glue_helper(E) vsock_loopback(E) snd_rawmidi(E) vmw_vsock_virtio_transport_common(E) snd_seq_device(E) ehci_pci(E) uhci_hcd(E) snd_pcm(E) ehci_hcd(E) vmw_balloon(E) ata_piix(E) snd_timer(E) snd(E) usbcore(E) joydev(E) pcspkr(E) libata(E) e1000(E)
[ 148.476005] vmw_vsock_vmci_transport(E) soundcore(E) i2c_piix4(E) vsock(E) sg(E) vmw_vmci(E) button(E) ac(E) msr(E) parport_pc(E) ppdev(E) lp(E) parport(E) fuse(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc32c_generic(E) crc16(E) mbcache(E) jbd2(E) sd_mod(E) t10_pi(E) crc_t10dif(E) crct10dif_generic(E) vmwgfx(E) drm_kms_helper(E) cec(E) ttm(E) drm(E) mptspi(E) scsi_transport_spi(E) mptscsih(E) mptbase(E) scsi_mod(E) psmouse(E) crct10dif_pclmul(E) crct10dif_common(E) evdev(E) crc32c_intel(E) serio_raw(E)
[ 148.499405] CPU: 2 PID: 2447 Comm: python3 Tainted: G OE 5.10.35+ #16
[ 148.500864] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 148.502789] RIP: 0010:__alloc_pages_nodemask+0x291/0x310
[ 148.503852] Code: 48 8b 04 25 c0 7b 01 00 48 05 48 0c 00 00 41 bd 01 00 00 00 48 89 44 24 08 e9 47 fe ff ff 81 e5 00 20 00 00 0f 85 60 ff ff ff <0f> 0b e9 59 ff ff ff 31 c0 e9 1b fe ff ff e8 4c 41 fc ff 48 89 c7
[ 148.507173] RSP: 0018:ffffb85fc04eca50 EFLAGS: 00010246
[ 148.508360] RAX: 0000000000000000 RBX: ffff9c2925286010 RCX: 0000000000000000
[ 148.510285] RDX: 0000000000000000 RSI: 000000000000000b RDI: 0000000000040a20
[ 148.512192] RBP: 0000000000000000 R08: 0000000000000001 R09: 00000000007ffff0
[ 148.513804] R10: 0000000000000020 R11: 0000000000000001 R12: 0000000000400010
[ 148.515284] R13: 000000000000000b R14: ffffb85fc04ecbe4 R15: ffff9c270db3e6c0
[ 148.516598] FS: 00007fe46910d700(0000) GS:ffff9c295a080000(0000) knlGS:0000000000000000
[ 148.518163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 148.519440] CR2: 00007fd854646ccc CR3: 00000001901ee002 CR4: 0000000000370ee0
[ 148.520741] Call Trace:
[ 148.521257] <IRQ>
[ 148.521654] __get_free_pages+0xd/0x30
[ 148.522372] __tfw_pool_realloc+0xd3/0x120 [tempesta_fw]
[ 148.523425] __str_grow_tree+0x45/0x140 [tempesta_fw]
[ 148.524340] __tfw_http_msg_add_str_data+0x38/0xd0 [tempesta_fw]
[ 148.525435] tfw_h2_parse_req+0x138/0x200 [tempesta_fw]
[ 148.526374] ss_skb_process+0xed/0x130 [tempesta_fw]
[ 148.527303] ? h2_set_hdr_if_mod_since+0x40/0x40 [tempesta_fw]
[ 148.528431] tfw_http_req_process+0x7c/0x870 [tempesta_fw]
[ 148.529439] ? ttls_decrypt+0x28e/0x510 [tempesta_tls]
[ 148.530424] ? ss_skb_process+0xed/0x130 [tempesta_fw]
[ 148.531359] ? tfw_h2_conn_terminate_close+0x30/0x30 [tempesta_fw]
[ 148.532947] tfw_h2_frame_process+0x26b/0x4b0 [tempesta_fw]
[ 148.534036] tfw_connection_recv+0x52/0xa0 [tempesta_fw]
[ 148.535215] tfw_tls_connection_recv+0x281/0x3a0 [tempesta_fw]
[ 148.537516] ss_tcp_process_data+0x1f1/0x400 [tempesta_fw]
[ 148.538632] ss_tcp_data_ready+0x3f/0xc0 [tempesta_fw]
[ 148.539933] tcp_rcv_established+0x5d8/0x680
[ 148.540933] tcp_v4_do_rcv+0x131/0x1f0
[ 148.541662] tcp_v4_rcv+0xc2f/0xd80
[ 148.542292] ip_protocol_deliver_rcu+0x2b/0x1b0
[ 148.543269] ip_local_deliver_finish+0x44/0x50
[ 148.544085] __netif_receive_skb_one_core+0x87/0xa0
[ 148.544958] process_backlog+0x96/0x160
[ 148.545668] net_rx_action+0x145/0x3e0
[ 148.546344] __do_softirq+0xcf/0x284
[ 148.546987] asm_call_irq_on_stack+0x12/0x20
[ 148.547879] </IRQ>
[ 148.548328] do_softirq_own_stack+0x37/0x40
[ 148.549087] do_softirq+0x5e/0x70
[ 148.549701] __local_bh_enable_ip+0x4b/0x50
[ 148.550527] ip_finish_output2+0x1ab/0x590
[ 148.551284] ? ipv4_link_failure+0x1b0/0x1b0
[ 148.552145] __ip_queue_xmit+0x180/0x410
[ 148.552911] __tcp_transmit_skb+0xa0e/0xbc0
[ 148.553661] tcp_write_xmit+0x401/0x1280
[ 148.554376] __tcp_push_pending_frames+0x32/0xf0
[ 148.555203] tcp_sendmsg_locked+0xa32/0xb50
[ 148.556040] tcp_sendmsg+0x28/0x40
[ 148.556697] sock_sendmsg+0x57/0x60
[ 148.557353] sock_write_iter+0x97/0x100
[ 148.558063] new_sync_write+0x199/0x1b0
[ 148.558756] vfs_write+0x1c2/0x260
[ 148.559536] ksys_write+0xa7/0xe0
[ 148.560242] ? exit_to_user_mode_prepare+0x6e/0x120
[ 148.561277] do_syscall_64+0x33/0x80
[ 148.562051] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 148.563334] RIP: 0033:0x7fe46dcfffef
[ 148.564021] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 29 fd ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 5c fd ff ff 48
[ 148.567729] RSP: 002b:00007fe46910a730 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[ 148.569246] RAX: ffffffffffffffda RBX: 000000000000001e RCX: 00007fe46dcfffef
[ 148.570624] RDX: 000000000000001e RSI: 00007fe46406eb73 RDI: 0000000000000005
[ 148.572108] RBP: 00007fe464003cd0 R08: 0000000000000000 R09: 00007fe46910a8b0
[ 148.573656] R10: 00007fe46cfb1d50 R11: 0000000000000293 R12: 00007fe46406eb73
[ 148.574995] R13: 000000000000001e R14: 00007fe464002808 R15: 00007fe464001f80
[ 148.576443] ---[ end trace 4c9ceb94de080a36 ]---
[ 148.577358] [tempesta fw] Warning: Cannot grow HTTP data string
[ 148.578784] [tempesta fw] Warning: failed to parse request: 127.0.0.1
Motivation
Currently, it is not possible to serve big responses (files) from backend, because of memory allocation error.
DMESG WARNING mm/page_alloc.c:4935
Related issue: https://github.com/tempesta-tech/tempesta/issues/498
Testing
To reproduce, backend on port 8000 should return a large response (tested on 1GB).
tempesta.cfg
curl -v http://127.0.0.1/test