kernel BUG at /root/tempesta-NEW/tempesta_fw/ss_skb.c:758

[keshonok]

This is the second time (in days, actually) that I have seen this bug. I don't have a specific test that would trigger it. Both times it happened in one of multiple tests runs using wrk. The bug occurs in the compatibility branch. So, the setup is like this:

• Linux kernel 4.1.27 for Tempesta. The FPU patch from the kernel 4.8.15 for Tempesta was backported to this version of the kernel. The patch from 4.8.15 that removed the use of the accept queue was NOT backported due to difficulty of doing that in 4.1.27. The changes are in the branch ab-4.8.15-backport of the kernel 4.1.27. The kernel is built with KASAN support.
• Tempesta code base. Current master was made compatible with the kernel 4.1.27 and with the patches to that kernel. The changes are in the branch ab-compat-4.1.27 of Tempesta. The changes include the manipulations with the accept queue the way it was before (plus a small bug fix).
• Both Tempesta and Nginx run in the same VM;
• Nginx runs with default configuration where is resets connections every 100 requests;
• Simplest Tempesta configuration:

  server 127.0.0.1:8080;

The configuration means that Tempesta bypasses cache on all requests. All requests are forwarded to the back-end server without exceptions.

The test is a simple wrk -t 8 -c 1024 -d 60m http://192.168.10.230 from the host machine to the VM with Tempesta and Nginx.

[167593.781686] ------------[ cut here ]------------
[167593.782250] kernel BUG at /root/tempesta-NEW/tempesta_fw/ss_skb.c:758!
[167593.782250] invalid opcode: 0000 [#1] SMP KASAN
[167593.784332] Modules linked in: tfw_sched_rr(O) tfw_sched_http(O) tfw_sched_hash(O) tempesta_fw(O) tempesta_db(O) tempesta_tls(O) aesni_intel aes_x86_64 glue_helper lrw gf128mul ppdev ablk_helper cryptd microcode 9pnet_virtio virtio_balloon i2c_piix4 i2c_core 9pnet pcspkr acpi_cpufreq parport_pc parport ip_tables ata_generic pata_acpi crc32c_intel serio_raw virtio_pci e1000 virtio_ring virtio ata_piix floppy ipv6 autofs4 [last unloaded: tempesta_tls]
[167593.784332] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G           O    4.1.27-ab+ #4
[167593.784332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[167593.784332] task: ffff8801d5adba50 ti: ffff8801d5b28000 task.ti: ffff8801d5b28000
[167593.784332] RIP: 0010:[<ffffffffa1d2afbb>]  [<ffffffffa1d2afbb>] __skb_fragment+0x1f4b/0x38d0 [tempesta_fw]
[167593.784332] RSP: 0018:ffff8801f6f06a48  EFLAGS: 00010282
[167593.784332] RAX: 00000000fffffff4 RBX: ffff88019ba37500 RCX: 0000000000000000
[167593.784332] RDX: 0000000000007f7e RSI: ffff88019b730000 RDI: 0000000000000082
[167593.784332] RBP: ffff8801f6f06ae8 R08: 0000000000000082 R09: ffff88019ba374c0
[167593.784332] R10: ffff880000000000 R11: 0000160000000000 R12: 0000000000000001
[167593.784332] R13: dffffc0000000000 R14: ffff88019ba3750c R15: ffff88019ba37000
[167593.784332] FS:  0000000000000000(0000) GS:ffff8801f6f00000(0000) knlGS:0000000000000000
[167593.784332] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[167593.784332] CR2: 00007f845f1d7000 CR3: 00000001d4242000 CR4: 00000000001406e0
[167593.784332] Stack:
[167593.784332]  ffff8801f6f06a78 0000000000000000 ffff8801f6f06a88 ffffffff813210e9
[167593.784332]  ffff8801d5b2ff58 0000000000000000 ffff880100000f4d ffff88019ba370e0
[167593.784332]  ffff88019ba37094 ffff88019ba37090 ffff88019ba370dc ffff880100000f4d
[167593.784332] Call Trace:
[167593.784332]  <IRQ> 
[167593.784332]  [<ffffffff813210e9>] ? is_ftrace_trampoline+0x99/0xe0
[167593.784332]  [<ffffffffa1d2d3af>] ss_skb_cutoff_data+0x2af/0x840 [tempesta_fw]
[167593.784332]  [<ffffffffa1d2d100>] ? ss_skb_get_room+0x240/0x240 [tempesta_fw]
[167593.784332]  [<ffffffff811ee13f>] ? mark_held_locks+0xcf/0x140
[167593.784332]  [<ffffffff814b743f>] ? cmpxchg_double_slab.isra.53+0x15f/0x180
[167593.784332]  [<ffffffffa1cde21d>] __hdr_del+0x11d/0x230 [tempesta_fw]
[167593.784332]  [<ffffffff821fc6a6>] ? _raw_spin_unlock_irqrestore+0x36/0x60
[167593.784332]  [<ffffffffa1ce1693>] tfw_http_msg_hdr_xfrm+0x773/0x8f0 [tempesta_fw]
[167593.784332]  [<ffffffffa1d29b9c>] ? __skb_fragment+0xb2c/0x38d0 [tempesta_fw]
[167593.784332]  [<ffffffffa1ce0f20>] ? tfw_http_msg_hdr_close+0x600/0x600 [tempesta_fw]
[167593.784332]  [<ffffffffa1d2d586>] ? ss_skb_cutoff_data+0x486/0x840 [tempesta_fw]
[167593.784332]  [<ffffffff814b97c5>] ? kmem_cache_free+0x315/0x410
[167593.784332]  [<ffffffffa1d2d100>] ? ss_skb_get_room+0x240/0x240 [tempesta_fw]
[167593.784332]  [<ffffffffa1cde2b0>] ? __hdr_del+0x1b0/0x230 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd004d>] tfw_http_set_hdr_connection+0x8d/0x1f0 [tempesta_fw]
[167593.784332]  [<ffffffffa1ce18f2>] ? tfw_http_msg_del_hbh_hdrs+0xe2/0x110 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd0670>] tfw_http_adjust_resp+0x110/0x3a0 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd0560>] ? tfw_http_conn_init+0xb0/0xb0 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd5850>] ? tfw_http_send_500.isra.27+0x130/0x130 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd5850>] ? tfw_http_send_500.isra.27+0x130/0x130 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd5873>] tfw_http_resp_cache_cb+0x23/0x1a0 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd0900>] ? tfw_http_adjust_resp+0x3a0/0x3a0 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd5850>] ? tfw_http_send_500.isra.27+0x130/0x130 [tempesta_fw]
[167593.784332]  [<ffffffffa1cc27de>] tfw_cache_add+0x1be/0x400 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd5850>] ? tfw_http_send_500.isra.27+0x130/0x130 [tempesta_fw]
[167593.784332]  [<ffffffffa1cc56f1>] tfw_cache_process+0x141/0x830 [tempesta_fw]
[167593.784332]  [<ffffffff8126a4c1>] ? __getnstimeofday64+0x81/0x1d0
[167593.784332]  [<ffffffffa1cc55b0>] ? tfw_cache_msg_cacheable+0x70/0x70 [tempesta_fw]
[167593.784332]  [<ffffffff811ee34e>] ? trace_hardirqs_on_caller+0x19e/0x5a0
[167593.784332]  [<ffffffff811ee75d>] ? trace_hardirqs_on+0xd/0x10
[167593.784332]  [<ffffffff8126a4f7>] ? __getnstimeofday64+0xb7/0x1d0
[167593.784332]  [<ffffffff8126a61e>] ? getnstimeofday64+0xe/0x30
[167593.784332]  [<ffffffffa1cd7089>] tfw_http_resp_cache+0x1d9/0x290 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd6eb0>] ? tfw_http_resp_gfsm+0x150/0x150 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd6e03>] ? tfw_http_resp_gfsm+0xa3/0x150 [tempesta_fw]
[167593.784332]  [<ffffffffa1cd859c>] tfw_http_msg_process+0xa8c/0x15d0 [tempesta_fw]
[167593.784332]  [<ffffffff811ee34e>] ? trace_hardirqs_on_caller+0x19e/0x5a0
[167593.784332]  [<ffffffff81e8eb1f>] ? __pg_skb_alloc+0x69f/0x9c0
[167593.784332]  [<ffffffffa1cd7b10>] ? tfw_http_conn_drop+0x9d0/0x9d0 [tempesta_fw]
[167593.784332]  [<ffffffff814be75c>] ? memset+0x2c/0x40
[167593.784332]  [<ffffffff81e8f1cd>] ? __alloc_skb+0x38d/0x750
[167593.784332]  [<ffffffff81e914e6>] ? skb_release_all+0x46/0x60
[167593.784332]  [<ffffffff81e91795>] ? consume_skb+0xf5/0x4f0
[167593.784332]  [<ffffffffa1cd7b10>] ? tfw_http_conn_drop+0x9d0/0x9d0 [tempesta_fw]
[167593.784332]  [<ffffffffa1cce73a>] __gfsm_fsm_exec+0x15a/0x2d0 [tempesta_fw]
[167593.784332]  [<ffffffffa1d2701f>] ? ss_skb_unroll+0x8bf/0xf60 [tempesta_fw]
[167593.784332]  [<ffffffffa1cceef1>] tfw_gfsm_dispatch+0x41/0x70 [tempesta_fw]
[167593.784332]  [<ffffffffa1cccefd>] tfw_connection_recv+0x1d/0x20 [tempesta_fw]
[167593.784332]  [<ffffffffa1d1b45b>] ss_tcp_process_data+0x56b/0xf40 [tempesta_fw]
[167593.784332]  [<ffffffff81191f75>] ? sched_clock_local+0x45/0x130
[167593.784332]  [<ffffffffa1d1aef0>] ? ss_synchronize+0x450/0x450 [tempesta_fw]
[167593.784332]  [<ffffffff81ff9990>] ? tcp_event_data_recv+0x550/0xc10
[167593.784332]  [<ffffffffa1d1cb4b>] ss_tcp_data_ready+0xeb/0x1a0 [tempesta_fw]
[167593.784332]  [<ffffffff82014871>] tcp_data_queue+0x10d1/0x47a0
[167593.784332]  [<ffffffff820137a0>] ? tcp_fastretrans_alert+0x2ff0/0x2ff0
[167593.784332]  [<ffffffff81ffbc50>] ? tcp_try_coalesce+0x550/0x550
[167593.784332]  [<ffffffff8201d807>] ? tcp_validate_incoming+0x757/0xf10
[167593.784332]  [<ffffffff8201e758>] tcp_rcv_established+0x798/0x2d40
[167593.784332]  [<ffffffff8201dfc0>] ? tcp_validate_incoming+0xf10/0xf10
[167593.784332]  [<ffffffff82054f89>] tcp_v4_do_rcv+0x419/0x1550
[167593.784332]  [<ffffffff8205a872>] tcp_v4_rcv+0x24b2/0x3fa0
[167593.784332]  [<ffffffffa1cc901f>] ? parse_cfg_entry+0x2cf/0xfa0 [tempesta_fw]
[167593.784332]  [<ffffffff820583c0>] ? tcp_v4_early_demux+0x860/0x860
[167593.784332]  [<ffffffff81fa571b>] ip_local_deliver_finish+0x30b/0xa40
[167593.784332]  [<ffffffff81fa558a>] ? ip_local_deliver_finish+0x17a/0xa40
[167593.784332]  [<ffffffff81f7ed00>] ? nf_ct_attach+0x1a0/0x1a0
[167593.784332]  [<ffffffff81fa6542>] ip_local_deliver+0x1e2/0x220
[167593.784332]  [<ffffffff81fa6360>] ? ip_call_ra_chain+0x510/0x510
[167593.784332]  [<ffffffff81f7f255>] ? nf_hook_slow+0x245/0x360
[167593.784332]  [<ffffffff81f7f015>] ? nf_hook_slow+0x5/0x360
[167593.784332]  [<ffffffff81fa4214>] ip_rcv_finish+0x4c4/0x16c0
[167593.784332]  [<ffffffff81fa6e7e>] ip_rcv+0x8fe/0x1320
[167593.784332]  [<ffffffff821a211f>] ? run_filter+0x11f/0x1f0
[167593.784332]  [<ffffffff81fa6580>] ? ip_local_deliver+0x220/0x220
[167593.784332]  [<ffffffff81fa3d50>] ? inet_del_offload+0x40/0x40
[167593.784332]  [<ffffffff81fa6580>] ? ip_local_deliver+0x220/0x220
[167593.784332]  [<ffffffff81ec9871>] __netif_receive_skb_core+0x771/0x2450
[167593.784332]  [<ffffffff81ec9100>] ? netif_set_xps_queue+0x15a0/0x15a0
[167593.784332]  [<ffffffff81ecb907>] ? process_backlog+0x217/0x650
[167593.784332]  [<ffffffff81ecb5aa>] __netif_receive_skb+0x5a/0x1a0
[167593.784332]  [<ffffffff811ee34e>] ? trace_hardirqs_on_caller+0x19e/0x5a0
[167593.784332]  [<ffffffff81ecb7ce>] process_backlog+0xde/0x650
[167593.784332]  [<ffffffff81ecb907>] ? process_backlog+0x217/0x650
[167593.784332]  [<ffffffff811acf0b>] ? enqueue_entity+0xc4b/0x4570
[167593.784332]  [<ffffffff81ed1a0f>] net_rx_action+0x5ef/0xfd0
[167593.784332]  [<ffffffff81ed1420>] ? napi_complete_done+0x1c0/0x1c0
[167593.784332]  [<ffffffff81108147>] __do_softirq+0x247/0xd30
[167593.784332]  [<ffffffff81108f15>] irq_exit+0x175/0x1a0
[167593.784332]  [<ffffffff8109c3c4>] smp_call_function_single_interrupt+0x74/0x90
[167593.784332]  [<ffffffff821fe220>] call_function_single_interrupt+0x70/0x80
[167593.784332]  <EOI> 
[167593.784332]  [<ffffffff821fc668>] ? _raw_spin_unlock_irq+0x38/0x40
[167593.784332]  [<ffffffff821fc660>] ? _raw_spin_unlock_irq+0x30/0x40
[167593.784332]  [<ffffffff81167c9f>] finish_task_switch+0x14f/0x270
[167593.784332]  [<ffffffff81167c73>] ? finish_task_switch+0x123/0x270
[167593.784332]  [<ffffffff821ecb7e>] __schedule+0x95e/0x1d20
[167593.784332]  [<ffffffff821edfd7>] schedule+0x97/0x1b0
[167593.784332]  [<ffffffff821ee845>] schedule_preempt_disabled+0x15/0x20
[167593.784332]  [<ffffffff811d5700>] cpu_startup_entry+0x240/0xaa0
[167593.784332]  [<ffffffff81279017>] ? clockevents_config_and_register+0x97/0xe0
[167593.784332]  [<ffffffff8109f0e6>] start_secondary+0x336/0x460
[167593.784332]  [<ffffffff8109edb0>] ? set_cpu_sibling_map+0x1c80/0x1c80
[167593.784332] Code: e8 00 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 16 0f 00 00 4d 8b ae e8 00 00 00 e9 37 fd ff ff <0f> 0b 48 8b 55 a8 48 b8 00 00 00 00 00 fc ff df 44 8b 6d b8 44 
[167593.784332] RIP  [<ffffffffa1d2afbb>] __skb_fragment+0x1f4b/0x38d0 [tempesta_fw]
[167593.784332]  RSP <ffff8801f6f06a48>

[krizhanovsky]

The bug has occurred on merged 4.1.27 with 4.8.15, so the merge bugs are very probable. Meantime, we do not support the kernel configuration and the bug has never appeared on maintained kernels. I suppose the bug is invalid and should be just closes. Now I mark it as low priority and probably will close it after #692.

[krizhanovsky]

Reproduced on current ak-692 branch:

    [ 1415.869792] ------------[ cut here ]------------
    [ 1415.872059] kernel BUG at /root/tempesta/tempesta_fw/ss_skb.c:758!
    [ 1415.872059] invalid opcode: 0000 [#1] SMP
    [ 1415.872059] Modules linked in: tfw_sched_ratio(O) tfw_sched_http(O) tfw_sched_hash(O) tempesta_fw(O) tempesta_db(O) tempesta_tls(O) crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper input_leds ablk_helper led_class bochs_drm cryptd ppdev serio_raw parport_pc ttm drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt sg parport button pcspkr ip_tables x_tables autofs4 ext4 crc16 jbd2 fscrypto mbcache sr_mod cdrom sd_mod ata_generic ata_piix libata i2c_piix4 crc32c_intel psmouse e1000 scsi_mod floppy
    [ 1415.893366] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           O    4.9.35 #3
    [ 1415.893366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
    [ 1415.893366] task: ffff91ef5d3bae40 task.stack: ffffb4414032c000
    [ 1415.893366] RIP: 0010:[<ffffffffc073b8dc>]  [<ffffffffc073b8dc>] __skb_fragment+0x9cc/0xbf0 [tempesta_fw]
    [ 1415.893366] RSP: 0018:ffffb4414032f760  EFLAGS: 00010282
    [ 1415.893366] RAX: 0000000000007fca RBX: ffff91ef3fc47ef0 RCX: 0000000000000030
    [ 1415.893366] RDX: 0000000000000036 RSI: 0000000000000000 RDI: 0000000000000036
    [ 1415.893366] RBP: 0000000000000000 R08: ffffb4414032f7c0 R09: 0000000000000003
    [ 1415.893366] R10: fffff5eb40000000 R11: ffff91ef00000000 R12: 0000000000000036
    [ 1415.893366] R13: ffff91ef3fc47ec0 R14: ffff91ef3fc47c00 R15: 00000000fffffff8
    [ 1415.893366] FS:  0000000000000000(0000) GS:ffff91ef7fc00000(0000) knlGS:0000000000000000
    [ 1415.893366] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [ 1415.893366] CR2: 00007fed04875000 CR3: 000000005c289000 CR4: 00000000003406f0
    [ 1415.893366] Stack:
    [ 1415.893366]  ffffb44100003e8c ffffb44100003e8c 000000000000001e ffff91ef427ac5e8
    [ 1415.893366]  0000000000000000 ffff91ef427ac3a0 ffff91ef427ac628 0000000000000002
    [ 1415.893366]  ffff91ef427ac038 ffffffffc073be2d 0000000000000002 0000000000000246
    [ 1415.893366] Call Trace:
    [ 1415.893366]  [<ffffffffc073be2d>] ? ss_skb_cutoff_data+0x11d/0x330 [tempesta_fw]
    [ 1415.893366]  [<ffffffff91729639>] ? _raw_spin_lock_irqsave+0x9/0x10
    [ 1415.893366]  [<ffffffff912b5c16>] ? lock_timer_base+0x76/0x90
    [ 1415.893366]  [<ffffffffc071fab0>] ? __hdr_del+0x60/0x110 [tempesta_fw]
    [ 1415.893366]  [<ffffffffc0720948>] ? tfw_http_msg_del_hbh_hdrs+0x38/0x50 [tempesta_fw]
    [ 1415.893366]  [<ffffffffc071b949>] ? tfw_http_adjust_resp+0x39/0x210 [tempesta_fw]
    [ 1415.893366]  [<ffffffffc0720337>] ? __tfw_http_msg_add_str_data+0x57/0x100 [tempesta_fw]
    [ 1415.893366]  [<ffffffffc071d3b0>] ? tfw_http_send_500.isra.28+0xe0/0xe0 [tempesta_fw]
    [ 1415.893366]  [<ffffffffc071d3c7>] ? tfw_http_resp_cache_cb+0x17/0x80 [tempesta_fw]
    [ 1415.893366]  [<ffffffffc0717f75>] ? tfw_cache_process+0x55/0x2a0 [tempesta_fw]
    [ 1415.893366]  [<ffffffffc071dcc2>] ? tfw_http_resp_cache+0x82/0xd0 [tempesta_fw]
    [ 1415.893366]  [<ffffffffc071e3da>] ? tfw_http_msg_process+0x4aa/0x760 [tempesta_fw]
    [ 1415.893366]  [<ffffffff91622d08>] ? __alloc_skb+0x38/0x1f0
    [ 1415.893366]  [<ffffffffc071af61>] ? __gfsm_fsm_exec+0x51/0x90 [tempesta_fw]
    [ 1415.893366]  [<ffffffffc0736f38>] ? ss_tcp_process_data+0x188/0x410 [tempesta_fw]
    [ 1415.893366]  [<ffffffffc073740e>] ? ss_tcp_data_ready+0x3e/0x80 [tempesta_fw]
    [ 1415.893366]  [<ffffffff9168b670>] ? tcp_rcv_established+0x460/0x6b0
    [ 1415.893366]  [<ffffffff9164dc28>] ? sk_filter_trim_cap+0x28/0x2b0
    [ 1415.893366]  [<ffffffff91695a1e>] ? tcp_v4_do_rcv+0x12e/0x1f0
    [ 1415.893366]  [<ffffffff91697192>] ? tcp_v4_rcv+0x8a2/0x9a0
    [ 1415.893366]  [<ffffffff9163788b>] ? __dev_queue_xmit+0x25b/0x620
    [ 1415.893366]  [<ffffffff91671474>] ? ip_local_deliver_finish+0x94/0x1c0
    [ 1415.893366]  [<ffffffff91671736>] ? ip_local_deliver+0x66/0xe0
    [ 1415.893366]  [<ffffffff9166a529>] ? nf_iterate+0x49/0x60
    [ 1415.893366]  [<ffffffff9166a5a8>] ? nf_hook_slow+0x68/0xc0
    [ 1415.893366]  [<ffffffff9161ab08>] ? __sk_dst_check+0x28/0x50
    [ 1415.893366]  [<ffffffff91671a31>] ? ip_rcv+0x281/0x3d0
    [ 1415.893366]  [<ffffffff91670ff0>] ? inet_del_offload+0x30/0x30
    [ 1415.893366]  [<ffffffff9163502a>] ? __netif_receive_skb_core+0x4aa/0x9b0
    [ 1415.893366]  [<ffffffff91692da0>] ? tcp_write_timer_handler+0x1e0/0x1e0
    [ 1415.893366]  [<ffffffff9161be3f>] ? sk_reset_timer+0xf/0x20
    [ 1415.893366]  [<ffffffff916364dd>] ? process_backlog+0x7d/0x110
    [ 1415.893366]  [<ffffffff91635cf1>] ? net_rx_action+0x201/0x2f0
    [ 1415.893366]  [<ffffffff9125ea23>] ? __do_softirq+0xf3/0x200
    [ 1415.893366]  [<ffffffff91279400>] ? sort_range+0x20/0x20
    [ 1415.893366]  [<ffffffff9125eb44>] ? run_ksoftirqd+0x14/0x30
    [ 1415.893366]  [<ffffffff912794f9>] ? smpboot_thread_fn+0xf9/0x150
    [ 1415.893366]  [<ffffffff91276112>] ? kthread+0xd2/0xf0
    [ 1415.893366]  [<ffffffff91276040>] ? kthread_park+0x50/0x50
    [ 1415.893366]  [<ffffffff91729b02>] ? ret_from_fork+0x22/0x30
    [ 1415.893366] Code: c5 48 83 c0 03 48 c1 e0 04 8b 44 02 08 48 c1 f9 06 48 c1 e1 0c 48 03 0d 2b ec 4e d1 4d 89 70 08 48 01 c1 49 89 08 e9 36 fa ff ff <0f> 0b 31 f6 ba 01 00 00 00 4c 89 f7 e8 13 eb ff ff 85 c0 44 8b 
    [ 1415.893366] RIP  [<ffffffffc073b8dc>] __skb_fragment+0x9cc/0xbf0 [tempesta_fw]
    [ 1415.893366]  RSP <ffffb4414032f760>

[krizhanovsky]

I believe the issue is duplicate of #692.

[keshonok]

I was getting a similar crash in ss_skb.c but in a different place.

[26922.687082] ------------[ cut here ]------------
[26922.687683] kernel BUG at /root/tempesta-NEW/tempesta_fw/ss_skb.c:319!
[26922.687795] invalid opcode: 0000 [#1] SMP KASAN
[26922.687795] Modules linked in: tfw_sched_ratio(O) tfw_sched_http(O) tfw_sched_hash(O) tempesta_fw(O) tempesta_db(O) tempesta_tls(O) ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul 9pnet_virtio ablk_helper 9pnet ppdev virtio_balloon cryptd pcspkr i2c_piix4 i2c_core input_leds led_class parport_pc parport ip_tables ata_generic pata_acpi crc32c_intel serio_raw virtio_pci virtio_ring e1000 virtio ata_piix floppy ipv6 crc_ccitt autofs4
[26922.687795] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W  O    4.9.35-ab+ #2
[26922.687795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[26922.687795] task: ffff88016749e040 task.stack: ffff8801674a8000
[26922.687795] RIP: 0010:[<ffffffffa09a087f>]  [<ffffffffa09a087f>] __extend_pgfrags+0x6bf/0xa40 [tempesta_fw]
[26922.687795] RSP: 0018:ffff880188e86c00  EFLAGS: 00010297
[26922.687795] RAX: 0000000000000011 RBX: ffff880160ddc058 RCX: ffffffffa09a0858
[26922.687795] RDX: 00000000ffffffff RSI: ffff880160ddc6f0 RDI: ffff880160ddc700
[26922.687795] RBP: ffff880188e86ca0 R08: ffff880160ddc000 R09: ffffea00056bd5c0
[26922.687795] R10: 0000000000000001 R11: ffffea00056bd5c0 R12: ffff880160ddc400
[26922.687795] R13: ffff880160ddc6c0 R14: ffff880160ddc2c0 R15: 0000000000000001
[26922.687795] FS:  0000000000000000(0000) GS:ffff880188e80000(0000) knlGS:0000000000000000
[26922.687795] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[26922.687795] CR2: 00007f0e4055a000 CR3: 000000015d996000 CR4: 00000000001406e0
[26922.687795] Stack:
[26922.687795]  ffff88015bd4c400 0000000002080020 ffffffff844d2200 ffffffff82b70064
[26922.687795]  0000000000000000 ffffffff82b70060 ffff880188e86ca0 ffffffff8150a91c
[26922.687795]  0000000000000400 ffff880160ddc4e0 ffff880160ddc4dc ffff880160ddc000
[26922.687795] Call Trace:
[26922.687795]  <IRQ>
[26922.687795]  [<ffffffff8150a91c>] ? alloc_pages_current+0x10c/0x430
[26922.687795]  [<ffffffffa09a34ba>] __new_pgfrag+0x61a/0x11c0 [tempesta_fw]
[26922.687795]  [<ffffffff8150a91c>] ? alloc_pages_current+0x10c/0x430
[26922.687795]  [<ffffffffa09a2ea0>] ? ss_skb_unroll+0xf40/0xf40 [tempesta_fw]
[26922.687795]  [<ffffffff8121c540>] ? debug_check_no_locks_freed+0x280/0x280
[26922.687795]  [<ffffffffa09a51ed>] __skb_fragment+0x118d/0x3920 [tempesta_fw]
[26922.687795]  [<ffffffff81ffadef>] ? enqueue_to_backlog+0x2ff/0xa50
[26922.687795]  [<ffffffff8121c01d>] ? trace_hardirqs_on+0xd/0x10
[26922.687795]  [<ffffffffa09a7f5f>] ss_skb_get_room+0xcf/0x230 [tempesta_fw]
[26922.687795]  [<ffffffff8121c540>] ? debug_check_no_locks_freed+0x280/0x280
[26922.687795]  [<ffffffffa0956428>] __hdr_expand+0x2b8/0x6f0 [tempesta_fw]
[26922.687795]  [<ffffffff81ffb8bf>] ? netif_rx_internal+0x37f/0x580
[26922.687795]  [<ffffffff81ffb7ee>] ? netif_rx_internal+0x2ae/0x580
[26922.687795]  [<ffffffff810c51a5>] ? default_send_IPI_single+0x75/0xa0
[26922.687795]  [<ffffffffa0956170>] ? __hdr_del+0x210/0x210 [tempesta_fw]
[26922.687795]  [<ffffffffa09591d7>] tfw_http_msg_hdr_xfrm+0x557/0x8c0 [tempesta_fw]
[26922.687795]  [<ffffffffa025c6ce>] ? tfw_sched_ratio_sched_sg_conn+0x3fe/0xfd0 [tfw_sched_ratio]
[26922.687795]  [<ffffffffa025c981>] ? tfw_sched_ratio_sched_sg_conn+0x6b1/0xfd0 [tfw_sched_ratio]
[26922.687795]  [<ffffffffa0958c80>] ? tfw_http_msg_hdr_close+0x5d0/0x5d0 [tempesta_fw]
[26922.687795]  [<ffffffffa025c9a0>] ? tfw_sched_ratio_sched_sg_conn+0x6d0/0xfd0 [tfw_sched_ratio]
[26922.687795]  [<ffffffff81e251c1>] ? loopback_xmit+0x161/0x330
[26922.687795]  [<ffffffffa09929dc>] ? tfw_sched_get_sg_srv_conn+0x11c/0x200 [tempesta_fw]
[26922.687795]  [<ffffffffa0992d15>] ? __tfw_sched_get_srv_conn+0x5/0x230 [tempesta_fw]
[26922.687795]  [<ffffffffa0992d15>] ? __tfw_sched_get_srv_conn+0x5/0x230 [tempesta_fw]
[26922.687795]  [<ffffffffa095329a>] tfw_http_req_cache_cb+0x18a/0x2b0 [tempesta_fw]
[26922.687795]  [<ffffffffa0953110>] ? tfw_http_conn_repair+0x3420/0x3420 [tempesta_fw]
[26922.687795]  [<ffffffffa0953110>] ? tfw_http_conn_repair+0x3420/0x3420 [tempesta_fw]
[26922.687795]  [<ffffffffa093e00e>] tfw_cache_process+0x2ce/0x7c0 [tempesta_fw]
[26922.687795]  [<ffffffffa093dd40>] ? tfw_cache_msg_cacheable+0x70/0x70 [tempesta_fw]
[26922.687795]  [<ffffffffa094f02c>] ? tfw_http_msg_process+0x87c/0x1540 [tempesta_fw]
[26922.687795]  [<ffffffffa094f03d>] tfw_http_msg_process+0x88d/0x1540 [tempesta_fw]
[26922.687795]  [<ffffffffa094e7b0>] ? tfw_http_conn_drop+0x9a0/0x9a0 [tempesta_fw]
[26922.687795]  [<ffffffff821001e0>] ? ip_output+0x1b0/0x520
[26922.687795]  [<ffffffff815261c1>] ? memset+0x31/0x40
[26922.687795]  [<ffffffff81fb5587>] ? __alloc_skb+0x397/0x6d0
[26922.687795]  [<ffffffffa09a0fb8>] ? ss_skb_queue_coalesce_tail+0x3b8/0x1360 [tempesta_fw]
[26922.687795]  [<ffffffff81fb7922>] ? skb_release_all+0x42/0x50
[26922.687795]  [<ffffffffa094e7b0>] ? tfw_http_conn_drop+0x9a0/0x9a0 [tempesta_fw]
[26922.687795]  [<ffffffffa0946d7e>] __gfsm_fsm_exec+0x14e/0x2c0 [tempesta_fw]
[26922.687795]  [<ffffffffa09a27fb>] ? ss_skb_unroll+0x89b/0xf40 [tempesta_fw]
[26922.687795]  [<ffffffffa0947511>] tfw_gfsm_dispatch+0x41/0x70 [tempesta_fw]
[26922.687795]  [<ffffffffa094573d>] tfw_connection_recv+0x1d/0x20 [tempesta_fw]
[26922.687795]  [<ffffffffa09965d1>] ss_tcp_process_data+0x551/0xed0 [tempesta_fw]
[26922.687795]  [<ffffffff811b6ad3>] ? sched_clock_local+0x43/0x120
[26922.687795]  [<ffffffffa0996080>] ? ss_synchronize+0x460/0x460 [tempesta_fw]
[26922.687795]  [<ffffffff82176808>] ? tcp_send_ack+0x458/0x640
[26922.687795]  [<ffffffff82138d91>] ? tcp_event_data_recv+0x4c1/0xe20
[26922.687795]  [<ffffffffa099814e>] ss_tcp_data_ready+0xbe/0x150 [tempesta_fw]
[26922.687795]  [<ffffffff8215be00>] tcp_rcv_established+0x1080/0x2dc0
[26922.687795]  [<ffffffff8204df32>] ? sk_filter_trim_cap+0x2a2/0x700
[26922.687795]  [<ffffffff8215ad80>] ? tcp_data_queue+0x4da0/0x4da0
[26922.687795]  [<ffffffff82194733>] ? tcp_v4_rcv+0x1f43/0x2f80
[26922.687795]  [<ffffffff8218ff29>] tcp_v4_do_rcv+0x529/0x8c0
[26922.687795]  [<ffffffff82195752>] tcp_v4_rcv+0x2f62/0x2f80
[26922.687795]  [<ffffffff8121c540>] ? debug_check_no_locks_freed+0x280/0x280
[26922.687795]  [<ffffffff820e370b>] ip_local_deliver_finish+0x2cb/0x9b0
[26922.687795]  [<ffffffff820e356a>] ? ip_local_deliver_finish+0x12a/0x9b0
[26922.687795]  [<ffffffff820e4680>] ip_local_deliver+0x380/0x450
[26922.687795]  [<ffffffff820e4495>] ? ip_local_deliver+0x195/0x450
[26922.687795]  [<ffffffff820e4300>] ? ip_call_ra_chain+0x510/0x510
[26922.687795]  [<ffffffff820bcb65>] ? nf_iterate+0x145/0x2d0
[26922.687795]  [<ffffffff820bcdd3>] ? nf_hook_slow+0xe3/0x190
[26922.687795]  [<ffffffff820e2169>] ip_rcv_finish+0x599/0x1870
[26922.687795]  [<ffffffff820e4f58>] ? ip_rcv+0x808/0x1300
[26922.687795]  [<ffffffff820e504d>] ? ip_rcv+0x8fd/0x1300
[26922.687795]  [<ffffffff820e5088>] ip_rcv+0x938/0x1300
[26922.687795]  [<ffffffff820e4f58>] ? ip_rcv+0x808/0x1300
[26922.687795]  [<ffffffff81280c80>] ? msleep+0xc0/0xc0
[26922.687795]  [<ffffffff820e4750>] ? ip_local_deliver+0x450/0x450
[26922.687795]  [<ffffffff820e1bd0>] ? inet_del_offload+0x40/0x40
[26922.687795]  [<ffffffff81f993e8>] ? sk_reset_timer+0x18/0x30
[26922.687795]  [<ffffffff820e4750>] ? ip_local_deliver+0x450/0x450
[26922.687795]  [<ffffffff81ff7fd0>] __netif_receive_skb_core+0x1690/0x2cf0
[26922.687795]  [<ffffffff8121c540>] ? debug_check_no_locks_freed+0x280/0x280
[26922.687795]  [<ffffffff8121c540>] ? debug_check_no_locks_freed+0x280/0x280
[26922.687795]  [<ffffffff82163c54>] ? tcp_established_options+0x94/0x3f0
[26922.687795]  [<ffffffff81ff6940>] ? net_tx_action+0x9d0/0x9d0
[26922.687795]  [<ffffffff81ff99be>] ? process_backlog+0x1fe/0x5f0
[26922.687795]  [<ffffffff81ff968a>] __netif_receive_skb+0x5a/0x190
[26922.687795]  [<ffffffff81ff9891>] process_backlog+0xd1/0x5f0
[26922.687795]  [<ffffffff81ff99be>] ? process_backlog+0x1fe/0x5f0
[26922.687795]  [<ffffffff81ffe925>] net_rx_action+0x655/0xde0
[26922.687795]  [<ffffffff81ffe2d0>] ? sk_busy_loop+0xba0/0xba0
[26922.687795]  [<ffffffff81ff61ca>] ? net_tx_action+0x25a/0x9d0
[26922.687795]  [<ffffffffa09981e0>] ? ss_tcp_data_ready+0x150/0x150 [tempesta_fw]
[26922.687795]  [<ffffffff81ff61f0>] ? net_tx_action+0x280/0x9d0
[26922.687795]  [<ffffffff81ff6150>] ? net_tx_action+0x1e0/0x9d0
[26922.687795]  [<ffffffff8112ecfd>] ? irq_exit+0x19d/0x1d0
[26922.687795]  [<ffffffff8232cae1>] __do_softirq+0x221/0x97d
[26922.687795]  [<ffffffff8112ecfd>] irq_exit+0x19d/0x1d0
[26922.687795]  [<ffffffff810b8350>] smp_call_function_single_interrupt+0x70/0x90
[26922.687795]  [<ffffffff8232ba89>] call_function_single_interrupt+0x89/0x90
[26922.687795]  <EOI>
[26922.687795]  [<ffffffff82327dc3>] ? default_idle+0x53/0x3b0
[26922.687795]  [<ffffffff82327dc1>] ? default_idle+0x51/0x3b0
[26922.687795]  [<ffffffff81074fbf>] arch_cpu_idle+0xf/0x20
[26922.687795]  [<ffffffff82328c5b>] default_idle_call+0x3b/0x60
[26922.687795]  [<ffffffff81200bb9>] cpu_startup_entry+0x269/0x310
[26922.687795]  [<ffffffff810bafe8>] start_secondary+0x2b8/0x3f0
[26922.687795]  [<ffffffff810bad30>] ? set_cpu_sibling_map+0x1d90/0x1d90
[26922.687795] Code: 78 59 b8 e0 41 0f b6 06 eb a9 31 f6 44 89 fa 4c 89 e7 4c 89 45 b8 e8 51 f9 ff ff 85 c0 4c 8b 45 b8 0f 84 22 fd ff ff eb 9b 0f 0b <0f> 0b 31 f6 e9 34 fe ff ff 4c 89 f7 4c 89 45 c0 e8 ec 67 b8 e0
[26922.687795] RIP  [<ffffffffa09a087f>] __extend_pgfrags+0x6bf/0xa40 [tempesta_fw]
[26922.687795]  RSP <ffff880188e86c00>