search

tempesta-tech / tempesta

All-in-one solution for high performance web content delivery and advanced protection against DDoS and web attacks
https://tempesta-tech.com/
GNU General Public License v2.0
617 stars 103 forks source link

tempesta-fw with KVM & TLS report a panic ! #1601

Open DKCopy opened 2 years ago

DKCopy commented 2 years ago

tempesta-fw with KVM & TLS report a panic !

version: tempesta-fw 0.6.8 ubuntu 20.04

KVM CPU: lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 40 bits physical, 48 bits virtual CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 8 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 61 Model name: Intel Core Processor (Broadwell) Stepping: 2 CPU MHz: 2600.000 BogoMIPS: 5200.00 Hypervisor vendor: KVM Virtualization type: full L1d cache: 256 KiB L1i cache: 256 KiB L2 cache: 32 MiB L3 cache: 128 MiB NUMA node0 CPU(s): 0-7 Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full generic retpoline, STIBP disabled, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_fr eq pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm cpuid_fault pti xsaveopt arat

tempesta config: access_log on; listen 80; listen 443 proto=https; block_action error reply; block_action attack drop; srv_group default { server 127.0.0.1:8080; } vhost default_vhost { sticky { cookie name=tfw_user_id enforce max_misses=10; } proxy_pass default; } srv_group sg2 { server 127.0.0.1:4443; } tls_match_any_server_name; tls_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem; tls_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;

vhost kangzy.tech { proxy_pass sg2; tls_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem; tls_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key; } cache 1; cache_fulfill ; http_chain { host == "*kangzy.tech" -> kangzy.tech;

-> block;

-> default_vhost;

} frang_limits { ip_block on; http_header_chunk_cnt 10; http_body_chunk_cnt 30; client_header_timeout 10; client_body_timeout 25; http_resp_code_block 401 403 100 10; } keepalive_timeout 50;

kernel message: Mar 26 17:46:02 192.168.122.120 [ 1034.129740] printk: console [netcon0] enabled Mar 26 17:46:02 192.168.122.120 [ 1034.131565] netconsole: network logging started Mar 26 17:46:07 192.168.122.120 [ 1039.445055] invalid opcode: 0000 [#1] SMP PTI Mar 26 17:46:07 192.168.122.120 [ 1039.445983] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Tainted: G OE 5.10.35+ #1 Mar 26 17:46:07 192.168.122.120 [ 1039.447573] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1.fc35 04/01/2014 Mar 26 17:46:07 192.168.122.120 [ 1039.449202] RIP: 0010:mpi_sqr_mont_mod_p256_x86_64+0x10/0x1c2 [tempesta_tls] Mar 26 17:46:07 192.168.122.120 [ 1039.450570] Code: 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 41 54 41 55 41 56 41 57 53 48 8b 16 4c 8b 7e 10 62 b3 f6 56 08 c4 62 a3 f6 66 18 4c 89 fa c4 e2 f3 f6 5e 08 c4 Mar 26 17:46:07 192.168.122.120 [ 1039.454045] RSP: 0018:ffffbb1040003430 EFLAGS: 00010206 Mar 26 17:46:07 192.168.122.120 [ 1039.455060] RAX: abe052fcb23e1145 RBX: ffffbb1040003620 RCX: eb4412097ebc9175 Mar 26 17:46:07 192.168.122.120 [ 1039.456460] RDX: b427f746fb2a033f RSI: ffffbb10400034a0 RDI: ffffbb10400034e0 Mar 26 17:46:07 192.168.122.120 [ 1039.457831] RBP: ffffbb10400035d0 R08: f49226e3ba9fe4d5 R09: 2b4a1e58c57a5c88 Mar 26 17:46:07 192.168.122.120 [ 1039.459215] R10: 0000000000000000 R11: 0000000000000000 R12: ffffbb1040003660 Mar 26 17:46:07 192.168.122.120 [ 1039.460603] R13: ffffbb1040003640 R14: 0000000000000023 R15: b459117160fc57e3 Mar 26 17:46:07 192.168.122.120 [ 1039.461978] FS: 0000000000000000(0000) GS:ffff98bcf7c00000(0000) knlGS:0000000000000000 Mar 26 17:46:07 192.168.122.120 [ 1039.463532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 26 17:46:07 192.168.122.120 [ 1039.464663] CR2: 00007ff0d396e180 CR3: 00000001229a6003 CR4: 00000000000606f0 Mar 26 17:46:07 192.168.122.120 [ 1039.466057] Call Trace: Mar 26 17:46:07 192.168.122.120 [ 1039.466594] Mar 26 17:46:07 192.168.122.120 [ 1039.467077] ? ecp256_mul_comb_core_g+0x31a/0x600 [tempesta_tls] Mar 26 17:46:07 192.168.122.120 [ 1039.468245] ecp256_mul_comb_g+0xa1/0x110 [tempesta_tls] Mar 26 17:46:07 192.168.122.120 [ 1039.469281] ecp256_gen_keypair+0x83/0x90 [tempesta_tls] Mar 26 17:46:07 192.168.122.120 [ 1039.470319] ttls_ecdh_make_params+0x48/0xc0 [tempesta_tls] Mar 26 17:46:07 192.168.122.120 [ 1039.471439] ? mpi_profile_clone+0x5a/0xb0 [tempesta_tls] Mar 26 17:46:07 192.168.122.120 [ 1039.472546] ttls_handshake_server_step+0x1416/0x1ab0 [tempesta_tls] Mar 26 17:46:07 192.168.122.120 [ 1039.473810] ? sha512_avx_update+0x15/0x20 [sha512_ssse3] Mar 26 17:46:07 192.168.122.120 [ 1039.474894] ? crypto_shash_update+0x1e/0x30 Mar 26 17:46:07 192.168.122.120 [ 1039.475755] ? ttls_update_checksum+0x6f/0x180 [tempesta_tls] Mar 26 17:46:07 192.168.122.120 [ 1039.476891] ttls_recv+0x3a4/0x6c0 [tempesta_tls] Mar 26 17:46:07 192.168.122.120 [ 1039.477889] ss_skb_process+0xf9/0x140 [tempesta_fw] Mar 26 17:46:07 192.168.122.120 [ 1039.478897] ? ttls_handle_alert+0x40/0x40 [tempesta_tls] Mar 26 17:46:07 192.168.122.120 [ 1039.479969] tfw_tls_msg_process+0xb9/0x3c0 [tempesta_fw] Mar 26 17:46:07 192.168.122.120 [ 1039.481048] ss_tcp_process_data+0x1e6/0x3f0 [tempesta_fw] Mar 26 17:46:07 192.168.122.120 [ 1039.482131] ss_tcp_data_ready+0x4e/0x90 [tempesta_fw] Mar 26 17:46:07 192.168.122.120 [ 1039.483168] tcp_data_ready+0x2b/0xd0 Mar 26 17:46:07 192.168.122.120 [ 1039.483938] tcp_data_queue+0x483/0xd20 Mar 26 17:46:07 192.168.122.120 [ 1039.484732] tcp_rcv_established+0x230/0x670 Mar 26 17:46:07 192.168.122.120 [ 1039.485564] ? sk_filter_trim_cap+0xde/0x240 Mar 26 17:46:07 192.168.122.120 [ 1039.486421] tcp_v4_do_rcv+0x140/0x200 Mar 26 17:46:07 192.168.122.120 [ 1039.487211] tcp_v4_rcv+0xcfd/0xe10 Mar 26 17:46:07 192.168.122.120 [ 1039.487938] ip_protocol_deliver_rcu+0x30/0x1b0 Mar 26 17:46:07 192.168.122.120 [ 1039.488854] ip_local_deliver_finish+0x48/0x60 Mar 26 17:46:07 192.168.122.120 [ 1039.489733] ip_local_deliver+0x72/0x110 Mar 26 17:46:07 192.168.122.120 [ 1039.490553] ? tcp_v4_early_demux+0xfa/0x160 Mar 26 17:46:07 192.168.122.120 [ 1039.491422] ? ip_rcv_finish_core.isra.0+0x146/0x420 Mar 26 17:46:07 192.168.122.120 [ 1039.492404] ip_rcv_finish+0x87/0xa0 Mar 26 17:46:07 192.168.122.120 [ 1039.493157] ip_rcv+0xcc/0xe0 Mar 26 17:46:07 192.168.122.120 [ 1039.493790] ? ip_rcv_finish_core.isra.0+0x420/0x420 Mar 26 17:46:07 192.168.122.120 [ 1039.494759] netif_receive_skb_one_core+0x88/0xa0 Mar 26 17:46:07 192.168.122.120 [ 1039.495723] netif_receive_skb+0x18/0x60 Mar 26 17:46:07 192.168.122.120 [ 1039.496562] process_backlog+0xa9/0x160 Mar 26 17:46:07 192.168.122.120 [ 1039.497350] net_rx_action+0x13e/0x390 Mar 26 17:46:07 192.168.122.120 [ 1039.498114] ? rcu_core+0xfa/0x500 Mar 26 17:46:07 192.168.122.120 [ 1039.498835] __do_softirq+0xea/0x2aa Mar 26 17:46:07 192.168.122.120 [ 1039.499503] asm_call_irq_on_stack+0x12/0x20 Mar 26 17:46:07 192.168.122.120 [ 1039.500232] Mar 26 17:46:07 192.168.122.120 [ 1039.500644] do_softirq_own_stack+0x3d/0x50 Mar 26 17:46:07 192.168.122.120 [ 1039.501364] irq_exit_rcu+0xa4/0xb0 Mar 26 17:46:07 192.168.122.120 [ 1039.501952] sysvec_call_function_single+0x3d/0x90 Mar 26 17:46:07 192.168.122.120 [ 1039.502738] asm_sysvec_call_function_single+0x12/0x20 Mar 26 17:46:07 fc34 xfce4-screensaver[17915]: Xlib: extension "DPMS" missing on display ":12.0". Mar 26 17:46:07 192.168.122.120 [ 1039.503616] RIP: 0010:native_safe_halt+0xe/0x10 Mar 26 17:46:07 192.168.122.120 [ 1039.504385] Code: 7b ff ff ff eb bd cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d 66 1b 43 00 f4 c3 66 90 e9 07 00 00 00 0f 00 2d 56 1b 43 00 fb f4 cc 0f 1f 44 00 00 55 48 89 e5 53 65 8b 15 df 69 43 70 0f 1f 44 Mar 26 17:46:07 192.168.122.120 [ 1039.507240] RSP: 0018:ffffffff90a03e10 EFLAGS: 00000216 Mar 26 17:46:07 192.168.122.120 [ 1039.508096] RAX: ffffffff8fbda9f0 RBX: 0000000000000000 RCX: ffff98bcf7c2cdc0 Mar 26 17:46:07 192.168.122.120 [ 1039.509210] RDX: 0000000000023a4e RSI: 0000000000000087 RDI: 0000000000000087 Mar 26 17:46:07 192.168.122.120 [ 1039.510356] RBP: ffffffff90a03e18 R08: ffff98bcf7c1f1e0 R09: 0000000000000000 Mar 26 17:46:07 192.168.122.120 [ 1039.511496] R10: 0000000000000001 R11: 0000000000000201 R12: ffffffff90a1a940 Mar 26 17:46:07 192.168.122.120 [ 1039.512638] R13: ffffffff90a1a940 R14: 0000000000000000 R15: 0000000000000000 Mar 26 17:46:07 192.168.122.120 [ 1039.513752] ? cpuidle_text_start+0x8/0x8 Mar 26 17:46:07 192.168.122.120 [ 1039.514455] ? default_idle+0xe/0x20 Mar 26 17:46:07 192.168.122.120 [ 1039.515095] arch_cpu_idle+0x15/0x20 Mar 26 17:46:07 192.168.122.120 [ 1039.515714] default_idle_call+0x38/0xc0 Mar 26 17:46:07 192.168.122.120 [ 1039.516366] do_idle+0x1f8/0x260 Mar 26 17:46:07 192.168.122.120 [ 1039.516926] cpu_startup_entry+0x20/0x30 Mar 26 17:46:07 192.168.122.120 [ 1039.517590] rest_init+0xb8/0xba Mar 26 17:46:07 192.168.122.120 [ 1039.518159] arch_call_rest_init+0xe/0x1b Mar 26 17:46:07 192.168.122.120 [ 1039.518835] start_kernel+0x84f/0x875 Mar 26 17:46:07 192.168.122.120 [ 1039.519463] x86_64_start_reservations+0x24/0x26 Mar 26 17:46:07 192.168.122.120 [ 1039.520219] x86_64_start_kernel+0x8b/0x8f Mar 26 17:46:07 192.168.122.120 [ 1039.520911] secondary_startup_64_no_verify+0xc2/0xcb Mar 26 17:46:07 192.168.122.120 [ 1039.521742] Modules linked in: netconsole tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 nls_utf8 isofs nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr intel_rapl_common rapl input_leds joydev serio_raw mac_hid qemu_fw_cfg sch_fq_codel ipmi_devintf ipmi_msghandler msr drm virtio_rng ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel i2c_i801 crypto_simd virtio_net ahci libahci cryptd glue_helper psmouse net_failover virtio_blk failover i2c_smbus lpc_ich xhci_pci xhci_pci_renesas [last unloaded: tempesta_lib]

krizhanovsky commented 2 years ago

Hi @DKCopy ,

thank you for your detailed report and using Tempesta FW!

The crash happens on

   1a3b0:       c4 62 b3 f6 56 08       mulx   0x8(%rsi),%r9,%r10

instruction, which is BMI2 extension. I do not see bmi2 in your cpu flags. Having that you have Broadwell and the CPU flags list is quite small, I'd assume that the VM wasn't started without -cpu host option, i.e. it doesn't use all the features of your CPU.

However, we do check the CPU for BMI2 in our Makefile and having that it seems you installed Tempesta FW from DKMS, your build should have failed... Or did you actually install (build) the package from outside the VM?

Probably we need to move all the CPU checks from the main Makefile to check_conf.pl and run it from tempesta.sh as well as from the Makefile.

DKCopy commented 2 years ago

Hi @DKCopy ,

thank you for your detailed report and using Tempesta FW!

The crash happens on

   1a3b0:       c4 62 b3 f6 56 08       mulx   0x8(%rsi),%r9,%r10

instruction, which is BMI2 extension. I do not see bmi2 in your cpu flags. Having that you have Broadwell and the CPU flags list is quite small, I'd assume that the VM wasn't started without -cpu host option, i.e. it doesn't use all the features of your CPU.

However, we do check the CPU for BMI2 in our Makefile and having that it seems you installed Tempesta FW from DKMS, your build should have failed... Or did you actually install (build) the package from outside the VM?

Probably we need to move all the CPU checks from the main Makefile to check_conf.pl and run it from tempesta.sh as well as from the Makefile.

Thanks a lot! I'll check it and report it again :)

DKCopy commented 2 years ago

Hi @DKCopy ,

thank you for your detailed report and using Tempesta FW!

The crash happens on

   1a3b0:       c4 62 b3 f6 56 08       mulx   0x8(%rsi),%r9,%r10

instruction, which is BMI2 extension. I do not see bmi2 in your cpu flags. Having that you have Broadwell and the CPU flags list is quite small, I'd assume that the VM wasn't started without -cpu host option, i.e. it doesn't use all the features of your CPU.

However, we do check the CPU for BMI2 in our Makefile and having that it seems you installed Tempesta FW from DKMS, your build should have failed... Or did you actually install (build) the package from outside the VM?

Probably we need to move all the CPU checks from the main Makefile to check_conf.pl and run it from tempesta.sh as well as from the Makefile.

Hi, krizhanovsky! I have test it with new platform that support BMI1&BMI2&AVX2 ! and have not occur kernel crash! :) thanks a lot!

krizhanovsky commented 2 years ago

Hi @DKCopy , you're welcome.

I move the task to development enhancement. We need to move all the CPU checks from the main Makefile to check_conf.pl and run it from tempesta.sh as well as from the Makefile.