search

tempesta-tech / tempesta

All-in-one solution for high performance web content delivery and advanced protection against DDoS and web attacks
https://tempesta-tech.com/
GNU General Public License v2.0
621 stars 103 forks source link

bug: kernel panic in `/root/tempesta/fw/ss_skb.c:1050!` #2231

Closed RomanBelozerov closed 2 months ago

RomanBelozerov commented 2 months ago

I receive kernel panic when I send request using curl curl -Ikvf -H "Host: jenkins" https://127.0.0.1:443/ TempestaFW - b13f4d16de7afabe03d1bec26d01b912e550fa64

I used local setup (the same IP): curl -> TempestaFW -> jenkins

Tempesta config

listen 80;
listen 443 proto=h2,https;

tls_certificate /tmp/tempesta/tempesta.crt;
tls_certificate_key /tmp/tempesta/tempesta.key;
tls_match_any_server_name;

server 192.168.122.116:8080; # Jenkins

curl log

root@ubuntu1:~/tempesta# curl -Ikvf -H "Host: jenkins" https://127.0.0.1:443/
*   Trying 127.0.0.1:443...
* Connected to 127.0.0.1 (127.0.0.1) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Tempesta Technologies Inc.; OU=Testing; CN=tempesta-tech.com; emailAddress=info@tempesta-tech.com
*  start date: Sep  2 16:29:35 2024 GMT
*  expire date: Sep  3 16:29:35 2025 GMT
*  issuer: C=US; ST=Washington; L=Seattle; O=Tempesta Technologies Inc.; OU=Testing; CN=tempesta-tech.com; emailAddress=info@tempesta-tech.com
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* h2h3 [:method: HEAD]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: jenkins]
* h2h3 [user-agent: curl/7.85.0-DEV]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x555568fb5160)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> HEAD / HTTP/2
> Host: jenkins
> user-agent: curl/7.85.0-DEV
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):

dmesg log

[  195.698695] ------------[ cut here ]------------
[  195.698725] kernel BUG at /root/tempesta/fw/ss_skb.c:1050!
[  195.698768] invalid opcode: 0000 [#1] SMP NOPTI
[  195.698791] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G           OE     5.10.35.tfw-14d2383 #1
[  195.698824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.15.0-0-g2dd4b9b3f840-prebuilt.qemu.org 04/01/2014
[  195.698891] RIP: 0010:ss_skb_cutoff_data+0x2ee/0x390 [tempesta_fw]
[  195.698916] Code: c0 85 c0 75 30 48 8b 45 d0 65 48 2b 04 25 28 00 00 00 0f 85 ae 00 00 00 48 83 c4 78 44 89 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b 41 89 c0 eb d4 0f 0b 0f 0b 48 c7 45 b0 00 00 00 00 8b 9d 68
[  195.698981] RSP: 0018:ffffbe4bc010cd30 EFLAGS: 00010246
[  195.699001] RAX: 0000000000000000 RBX: ffff9a71b62211c0 RCX: 0000000000000000
[  195.699027] RDX: 0000000000000000 RSI: ffff9a71b62211c0 RDI: 0000000000000000
[  195.699053] RBP: ffffbe4bc010cdd0 R08: ffffffffc0aa15d1 R09: ffff9a71b6221010
[  195.699079] R10: 0000000000000000 R11: ffff9a71b6221020 R12: ffff9a71a888f0c0
[  195.699105] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  195.699133] FS:  0000000000000000(0000) GS:ffff9a72b7d00000(0000) knlGS:0000000000000000
[  195.699162] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  195.699184] CR2: 00005632acf06000 CR3: 00000001288f8005 CR4: 0000000000770ee0
[  195.699212] PKRU: 55555554
[  195.699224] Call Trace:
[  195.699239]  <IRQ>
[  195.699264]  tfw_http_msg_cutoff_body_chunks+0x35/0x80 [tempesta_fw]
[  195.699298]  tfw_h2_stream_xmit_prepare_resp+0x169/0x180 [tempesta_fw]
[  195.699332]  tfw_h2_make_frames+0x2b3/0x5a0 [tempesta_fw]
[  195.699363]  tfw_sk_fill_write_queue+0x150/0x1d0 [tempesta_fw]
[  195.700299]  ? tcp_current_mss+0x5e/0xb0
[  195.701213]  ss_tx_action+0x6a6/0x770 [tempesta_fw]
[  195.702126]  net_tx_action+0x9c/0x250
[  195.703017]  __do_softirq+0xd9/0x291
[  195.703872]  asm_call_irq_on_stack+0x12/0x20
[  195.704708]  </IRQ>
[  195.705556]  do_softirq_own_stack+0x3d/0x50
[  195.706384]  irq_exit_rcu+0xa4/0xb0
[  195.707196]  common_interrupt+0x7d/0x150
[  195.708004]  asm_common_interrupt+0x1e/0x40
[  195.708803] RIP: 0010:native_safe_halt+0xe/0x10
[  195.709610] Code: 7b ff ff ff eb bd cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d c6 9e 47 00 f4 c3 66 90 e9 07 00 00 00 0f 00 2d b6 9e 47 00 fb f4 <c3> cc 0f 1f 44 00 00 55 48 89 e5 53 65 8b 15 3f ed 67 79 0f 1f 44
[  195.711247] RSP: 0018:ffffbe4bc0093e90 EFLAGS: 00000206
[  195.712068] RAX: ffffffff86992690 RBX: 0000000000000002 RCX: ffff9a72b7d2cdc0
[  195.712893] RDX: 000000000000edd6 RSI: 0000000000000082 RDI: 0000000000000082
[  195.713711] RBP: ffffbe4bc0093e98 R08: 0000000000000000 R09: 0000000000000013
[  195.714528] R10: 0000000000000002 R11: 0000000000000012 R12: ffff9a71a02d4800
[  195.715350] R13: ffff9a71a02d4800 R14: 0000000000000000 R15: 0000000000000000
[  195.716164]  ? __cpuidle_text_start+0x8/0x8
[  195.716970]  ? default_idle+0xe/0x20
[  195.717783]  arch_cpu_idle+0x15/0x20
[  195.718568]  default_idle_call+0x38/0xc0
[  195.719346]  do_idle+0x1f8/0x260
[  195.720112]  cpu_startup_entry+0x20/0x30
[  195.720879]  start_secondary+0x111/0x150
[  195.721644]  secondary_startup_64_no_verify+0xb0/0xbb
[  195.722383] Modules linked in: tls tempesta_fw(OE) tempesta_db(OE) sha256_ssse3 sha512_ssse3 tempesta_tls(OE) tempesta_lib(OE) tcp_diag inet_diag binfmt_misc kvm_amd ccp kvm joydev input_leds mac_hid qemu_fw_cfg serio_raw dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua sch_fq_codel efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel bochs_drm drm_vram_helper aesni_intel drm_ttm_helper ttm drm_kms_helper psmouse syscopyarea sysfillrect crypto_simd sysimgblt fb_sys_fops virtio_scsi cryptd cec glue_helper drm virtio_net net_failover failover i2c_piix4 pata_acpi floppy
[  195.727948] ---[ end trace a8c1d70249e7b191 ]---
[  195.728803] RIP: 0010:ss_skb_cutoff_data+0x2ee/0x390 [tempesta_fw]
[  195.729637] Code: c0 85 c0 75 30 48 8b 45 d0 65 48 2b 04 25 28 00 00 00 0f 85 ae 00 00 00 48 83 c4 78 44 89 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b 41 89 c0 eb d4 0f 0b 0f 0b 48 c7 45 b0 00 00 00 00 8b 9d 68
[  195.731361] RSP: 0018:ffffbe4bc010cd30 EFLAGS: 00010246
[  195.732222] RAX: 0000000000000000 RBX: ffff9a71b62211c0 RCX: 0000000000000000
[  195.733121] RDX: 0000000000000000 RSI: ffff9a71b62211c0 RDI: 0000000000000000
[  195.733973] RBP: ffffbe4bc010cdd0 R08: ffffffffc0aa15d1 R09: ffff9a71b6221010
[  195.734838] R10: 0000000000000000 R11: ffff9a71b6221020 R12: ffff9a71a888f0c0
[  195.735709] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  195.736578] FS:  0000000000000000(0000) GS:ffff9a72b7d00000(0000) knlGS:0000000000000000
[  195.737481] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

Jenkins response for curl -Ikvf -H "Host: jenkins" https://127.0.0.1:443/

   Trying 192.168.122.116:80...
* Connected to 192.168.122.116 (192.168.122.116) port 80 (#0)
> HEAD / HTTP/1.1
> Host: jenkins
> User-Agent: curl/7.85.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
HTTP/1.1 403 Forbidden
< Date: Tue, 03 Sep 2024 13:14:42 GMT
Date: Tue, 03 Sep 2024 13:14:42 GMT
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Set-Cookie: JSESSIONID.dd9ab07a=node0brxereqvxjtvm8k0qdd6stca2.node0; Path=/; HttpOnly
Set-Cookie: JSESSIONID.dd9ab07a=node0brxereqvxjtvm8k0qdd6stca2.node0; Path=/; HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Content-Type: text/html;charset=utf-8
Content-Type: text/html;charset=utf-8
< X-Hudson: 1.395
X-Hudson: 1.395
< X-Jenkins: 2.462.1
X-Jenkins: 2.462.1
< X-Jenkins-Session: 73acbb65
X-Jenkins-Session: 73acbb65
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< via: 1.1 tempesta_fw (Tempesta FW 0.8.0)
via: 1.1 tempesta_fw (Tempesta FW 0.8.0)
< Server: Tempesta FW/0.8.0
Server: Tempesta FW/0.8.0

* The requested URL returned error: 403
* Closing connection 0
curl: (22) The requested URL returned error: 403