search

tempesta-tech / tempesta

All-in-one solution for high performance web content delivery and advanced protection against DDoS and web attacks
https://tempesta-tech.com/
GNU General Public License v2.0
621 stars 103 forks source link

BUG when caching chunked response #1934

Closed RomanBelozerov closed 1 year ago

RomanBelozerov commented 1 year ago

Scope

My config (Tempesta on separate VM) :

[General]
ip = 192.168.122.1
ipv6 = ::1
verbose = 6
workdir = /tmp/host
duration = 10
concurrent_connections = 10
log_file = tests_log.log
stress_threads = 2
stress_large_content_length = 65536
stress_requests_count = 100
stress_mtu = 1500
long_body_size = 500

[Client]
ip = 192.168.122.1
ipv6 = ::1
hostname = localhost
ab = ab
wrk = wrk
h2load = h2load
tls-perf = tls-perf
workdir = /tmp/client
unavaliable_timeout = 300

[Tempesta]
ip = 192.168.122.116
ipv6 = ::1
hostname = 192.168.122.116
user = root
port = 22
srcdir = /root/tempesta
workdir = /tmp/host
config = tempesta.conf
tmp_config = tempesta_tmp.conf
unavaliable_timeout = 300

[Server]
ip = 192.168.122.1
ipv6 = ::1
hostname = localhost
user = root
port = 22
nginx = nginx
workdir = /tmp/nginx
resources = /var/www/html/
aliases_interface = virbr0
aliases_base_ip = 192.168.122.1
max_workers = 16
keepalive_timeout = 60
keepalive_requests = 100
unavaliable_timeout = 300

[  720.093923] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [swapper/2:0]
[  720.094682] Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat br_netfilter bridge stp llc snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_hda_codec intel_rapl_msr overlay snd_hda_core binfmt_misc snd_hwdep intel_rapl_common snd_pcm kvm_intel kvm snd_seq_midi snd_seq_midi_event qxl snd_rawmidi drm_ttm_helper snd_seq ttm crct10dif_pclmul ghash_clmulni_intel snd_seq_device aesni_intel snd_timer crypto_simd cryptd drm_kms_helper glue_helper nls_iso8859_1 snd joydev input_leds cec serio_raw rc_core soundcore fb_sys_fops syscopyarea sysfillrect sysimgblt mac_hid qemu_fw_cfg sch_fq_codel drm msr parport_pc ppdev lp parport ramoops reed_solomon pstore_blk pstore_zone efi_pstore virtio_rng ip_tables x_tables autofs4 hid_generic usbhid hid virtio_net
[  720.094729]  psmouse net_failover crc32_pclmul ahci failover libahci i2c_i801 xhci_pci i2c_smbus virtio_blk lpc_ich xhci_pci_renesas [last unloaded: tempesta_lib]
[  720.101717] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G           OE     5.10.35+ #1
[  720.102201] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1 04/01/2014
[  720.102708] RIP: 0010:tfw_cache_copy_resp.isra.0+0x1300/0x20f0 [tempesta_fw]
[  720.103167] Code: 48 c7 85 60 ff ff ff 00 00 00 00 48 89 85 48 ff ff ff 48 89 95 50 ff ff ff 48 85 d2 75 4a 4c 89 d0 48 39 c3 0f 84 ac 00 00 00 <4c> 39 e8 75 a9 41 83 c4 01 45 3b a1 c0 00 00 00 0f 83 fa 03 00 00
[  720.104364] RSP: 0018:ffffb0c80011c580 EFLAGS: 00000297
[  720.104691] RAX: ffff935d2bc49905 RBX: ffff935d2a4a11c0 RCX: 0000000000000034
[  720.105134] RDX: ffff935d22823ce8 RSI: 0000000000000000 RDI: 0000000000000182
[  720.105586] RBP: ffffb0c80011c790 R08: ffff935e77cafac0 R09: ffff935d2b00e538
[  720.106033] R10: ffff935d0f20d193 R11: ffff935d0f20d010 R12: 0000000000000001
[  720.106464] R13: ffff935d2bc49915 R14: 0000000000000006 R15: ffff935d2a4a1182
[  720.106895] FS:  0000000000000000(0000) GS:ffff935e77c80000(0000) knlGS:0000000000000000
[  720.107408] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  720.107794] CR2: 00007fb12b367ae0 CR3: 0000000269e10002 CR4: 0000000000770ee0
[  720.108228] PKRU: 55555554
[  720.108398] Call Trace:
[  720.108553]  <IRQ>
[  720.108684]  ? virtqueue_add_split+0x131/0x510
[  720.108965]  __cache_add_node+0xd1/0x110 [tempesta_fw]
[  720.109282]  tfw_cache_do_action+0xa2e/0x1260 [tempesta_fw]
[  720.109626]  ? tfw_h2_resp_adjust_fwd+0x780/0x780 [tempesta_fw]
[  720.110000]  ? frang_resp_handler+0xcc/0x410 [tempesta_fw]
[  720.110341]  ? frang_resp_handler+0xcc/0x410 [tempesta_fw]
[  720.110680]  ? tfw_str_add_compound+0x1b/0x30 [tempesta_fw]
[  720.111080]  ? __resp_parse_body+0x2b1/0x800 [tempesta_fw]
[  720.111426]  ? tfw_h2_resp_adjust_fwd+0x780/0x780 [tempesta_fw]
[  720.111866]  tfw_cache_process+0xc9/0x360 [tempesta_fw]
[  720.112204]  ? tfw_cache_process+0xc9/0x360 [tempesta_fw]
[  720.112557]  tfw_http_resp_cache+0xf7/0x1a0 [tempesta_fw]
[  720.112910]  tfw_http_msg_process_generic+0x626/0x740 [tempesta_fw]
[  720.113310]  ? kmem_cache_free+0x105/0x470
[  720.113577]  tfw_http_msg_process+0x38/0x50 [tempesta_fw]
[  720.113930]  tfw_connection_recv+0x56/0xa0 [tempesta_fw]
[  720.114278]  ss_tcp_process_data+0x1e0/0x430 [tempesta_fw]
[  720.114636]  ss_tcp_data_ready+0x44/0xf0 [tempesta_fw]
[  720.114967]  tcp_data_ready+0x2b/0xd0
[  720.115213]  tcp_rcv_established+0x530/0x6b0
[  720.115492]  tcp_v4_do_rcv+0x140/0x200
[  720.115756]  tcp_v4_rcv+0xcd0/0xe20
[  720.115983]  ip_protocol_deliver_rcu+0x44/0x230
[  720.116275]  ip_local_deliver_finish+0x48/0x60
[  720.116561]  ip_local_deliver+0xf8/0x110
[  720.116815]  ? ip_protocol_deliver_rcu+0x230/0x230
[  720.117123]  ip_rcv_finish+0x87/0xa0
[  720.117355]  ip_rcv+0xce/0xe0
[  720.117550]  ? ip_rcv_finish_core.constprop.0+0x470/0x470
[  720.117897]  __netif_receive_skb_core+0x4ed/0x1030
[  720.118212]  ? consume_skb+0x46/0xb0
[  720.118445]  ? packet_rcv+0x54/0x490
[  720.118679]  __netif_receive_skb_one_core+0x3f/0xa0
[  720.118999]  __netif_receive_skb+0x18/0x60
[  720.119264]  process_backlog+0x9e/0x170
[  720.119504]  net_rx_action+0x13b/0x430
[  720.119772]  __do_softirq+0xe3/0x340
[  720.119994]  asm_call_irq_on_stack+0xf/0x20
[  720.120251]  </IRQ>
[  720.120388]  do_softirq_own_stack+0x3d/0x50
[  720.120652]  irq_exit_rcu+0xa2/0xe0
[  720.120873]  sysvec_call_function_single+0x3d/0x90
[  720.121173]  asm_sysvec_call_function_single+0x12/0x20
[  720.121495] RIP: 0010:native_safe_halt+0xe/0x10
[  720.121778] Code: 39 ff ff ff 4c 89 ee 48 c7 c7 a0 ba 45 98 e8 19 7b 8e ff e9 01 ff ff ff cc cc cc cc e9 07 00 00 00 0f 00 2d 66 a8 44 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 56 a8 44 00 f4 c3 cc cc 0f 1f 44 00
[  720.122938] RSP: 0018:ffffb0c800093e88 EFLAGS: 00000202
[  720.123284] RAX: ffffffff975c02c0 RBX: 0000000000000002 RCX: ffff935e77cace40
[  720.123757] RDX: 00000000000f4862 RSI: ffffb0c800093e28 RDI: 000000a134ce03b0
[  720.124200] RBP: ffffb0c800093e90 R08: 0000000000000001 R09: 0000000000000221
[  720.124635] R10: 0000000000000003 R11: 0000000000000000 R12: ffff935d202ebc80
[  720.125069] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  720.125505]  ? __sched_text_end+0x4/0x4
[  720.125743]  ? default_idle+0xe/0x20
[  720.125974]  arch_cpu_idle+0x15/0x20
[  720.126201]  default_idle_call+0x3d/0xc0
[  720.126450]  do_idle+0x215/0x2a0
[  720.126656]  cpu_startup_entry+0x20/0x30
[  720.126905]  start_secondary+0x145/0x1b0
[  720.127178]  secondary_startup_64_no_verify+0xc2/0xcb

Testing

krizhanovsky commented 1 year ago

I can not reporduce the issue in my VM, but while the webserver and my VM use the same Ubuntu 22 Server, the Tempesta build on my VM inlines tfw_cache_h2_copy_chunked_body() into tfw_cache_copy_resp(), while the server doesn't. The issue is likely a duplicate of https://github.com/tempesta-tech/tempesta/issues/1970

krizhanovsky commented 1 year ago

Fixed in #1972