Closed vankoven closed 5 years ago
Error at test_big_body_to_chunked (msg_modification.tests_encode_to_chunked.TestTransformPayload)
is a known issue: tempesta-tech/tempesta-test#92 .
The other issues are the same assertion on teardown: Warnings happened during test on Tempesta
. This means that Tempesta or kernel throws a warning (WARN_ON()
macro) during work. I bet it's the same issue as in unittests, since the trace looks absolutely the same.
Here is an original trace log:
[ 152.801791] tfw_test: === request: [GET / HTTP/1.1
Host: test
Connection: close, "foo"
]
[ 152.807236] ------------[ cut here ]------------
[ 152.808248] WARNING: CPU: 1 PID: 4256 at /home/user/qtc/release/tempesta/tempesta_fw/t/unit/../../..//tempesta_fw/http_parser.c:1212
__parse_connection+0x337/0x660 [tfw_test]
[ 152.813157] Modules linked in: tfw_test(O+) tfw_fuzzer(O) tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3
sha512_ssse3 sha512_generic ccm ctr gcm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic ghash_clmulni_inte
l iTCO_wdt binfmt_misc iTCO_vendor_support snd_hda_intel snd_hda_codec qxl evdev snd_hda_core ttm snd_hwdep serio_raw pcspkr lpc_ich sn
d_pcm drm_kms_helper snd_timer snd virtio_balloon sg virtio_console soundcore mfd_core shpchp drm button ip_tables x_tables autofs4 ext
4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb aesni_intel crypto_simd cryptd(O) glue_helper aes_x86_64 sr_mod cdrom virtio_blk virti
o_net ahci libahci crc32c_intel i2c_i801 libata uhci_hcd ehci_pci ehci_hcd psmouse sym53c8xx scsi_transport_spi usbcore usb_common scsi
_mod
[ 152.834485] virtio_pci virtio_ring virtio [last unloaded: tempesta_lib]
[ 152.836335] CPU: 1 PID: 4256 Comm: insmod Tainted: G O 4.14.0-tempesta-amd64 #1 Debian 4.14.32-tfw5-1
[ 152.840026] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-20181126_142135-anatol 04/01/2014
[ 152.842680] task: ffff8cad1ad31e00 task.stack: ffffa62580668000
[ 152.845041] RIP: 0010:__parse_connection+0x337/0x660 [tfw_test]
[ 152.847462] RSP: 0018:ffffa6258066ba90 EFLAGS: 00010212
[ 152.849052] RAX: 000000000000002c RBX: ffffffffc0dbd005 RCX: 0000000000000005
[ 152.851376] RDX: 0000000000000000 RSI: ffffffffc0db0c54 RDI: ffffffffc0dbd000
[ 152.853179] RBP: ffffffffc102b300 R08: ffff8cad17966800 R09: 0000000000000460
[ 152.854822] R10: 00000000000001c0 R11: 0000000000000005 R12: ffff8cad1b1dd020
[ 152.856972] R13: 0000000000000010 R14: 0000000000000010 R15: ffffffffc0dbd000
[ 152.859533] FS: 00007fa179be0540(0000) GS:ffff8cad3fd00000(0000) knlGS:0000000000000000
[ 152.861261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 152.863658] CR2: 000055a92d9ea518 CR3: 0000000057974004 CR4: 00000000003606e0
[ 152.866167] Call Trace:
[ 152.867076] ? tfw_http_parse_req+0x6839/0xbb20 [tfw_test]
[ 152.868498] ? test_req_alloc+0x7c/0xa0 [tfw_test]
[ 152.870499] ? do_split_and_parse+0x111/0x176 [tfw_test]
[ 152.872196] ? test__http_parser__mangled_messages+0x209/0xabf [tfw_test]
[ 152.874637] ? test_suite__http_parser+0x5047/0x84a0 [tfw_test]
[ 152.876014] ? vprintk_emit+0x2fc/0x3a0
[ 152.877615] ? 0xffffffffc0991000
[ 152.879021] ? printk+0x52/0x6e
[ 152.880020] ? __bzero_test+0x58/0x160 [tfw_test]
[ 152.881934] ? 0xffffffffc0991000
[ 152.883168] ? test_run_all+0x17b/0x3a0 [tfw_test]
[ 152.884405] ? test_run_all+0x17b/0x3a0 [tfw_test]
[ 152.886058] ? tfw_test_init+0x14/0x1000 [tfw_test]
[ 152.887132] ? 0xffffffffc0991000
[ 152.888094] ? do_one_initcall+0x51/0x190
[ 152.889755] ? kmem_cache_alloc_trace+0xe2/0x1a0
[ 152.891452] ? do_init_module+0x22/0x1f8
[ 152.892278] ? do_init_module+0x5b/0x1f8
[ 152.893705] ? load_module+0x25fc/0x2d40
[ 152.895269] ? vfs_read+0x119/0x130
[ 152.896261] ? SYSC_finit_module+0xd2/0x100
[ 152.897516] ? SYSC_finit_module+0xd2/0x100
[ 152.899229] ? do_syscall_64+0x8d/0x120
[ 152.900129] ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ 152.901346] Code: 00 00 48 c7 45 78 00 00 00 00 48 89 d8 48 c7 85 80 00 00 00 00 00 00 00 48 c7 45 48 10 2a d4 c0 4c 29 f8 49 39 c5 76 ab 0f b6 03 <0f> 0b 3c 2c 77 2b 48 ba 00 02 00 00 01 10 00 00 48 0f a3 c2 0f
[ 152.906529] ---[ end trace 84e319afbae2f3c7 ]---
And from functional tests:
[ 1851.989328] Start test: reconf.test_stress_sched_ratio.SchedRatioStaticCustomSg.test_ratio_del_srvs
[ 1854.526340] [tdb] Start Tempesta DB
[ 1854.540849] net_ratelimit: 620 callbacks suppressed
[ 1854.540849] [tempesta fw] Initializing Tempesta FW kernel module...
[ 1854.546259] [tempesta fw] Registering new classifier: frang
[ 1854.549728] [tempesta fw] Registering new scheduler: hash
[ 1854.552284] [tempesta fw] Registering new scheduler: ratio
[ 1854.593571] [tempesta fw] Preparing for the configuration processing.
[ 1854.596458] [tempesta fw] Configuration processing is completed.
[ 1854.606210] [tdb] Opened table /opt/tempesta/db/filter.tdb: size=16777216 rec_size=20 base=ffff99b62ac00000
[ 1854.751668] [tdb] Opened table /opt/tempesta/db/cache.tdb: size=268435456 rec_size=0 base=ffff99b61ac00000
[ 1854.762778] [tdb] Opened table /opt/tempesta/db/client.tdb: size=16777216 rec_size=616 base=ffff99b619c00000
[ 1854.767037] [tempesta fw] Open listen socket on: 0.0.0.0
[ 1854.784018] [tempesta fw] modules are started
[ 1856.943275] [tempesta fw] Live reconfiguration of Tempesta.
[ 1856.946306] [tempesta fw] Preparing for the configuration processing.
[ 1858.002750] ------------[ cut here ]------------
[ 1858.005034] WARNING: CPU: 0 PID: 0 at /root/tempesta/tempesta/tempesta_fw/http_parser.c:1212 __parse_connection+0x5a3/0x680 [tempesta_fw]
[ 1858.010271] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) iptable_mangle xt_mark tcp_diag inet_diag sha256_ssse3 sha512_ssse3 sha512_generic ccm ctr ghash_generic gf128mul gcm fuse ata_generic intel_rapl cirrus sb_edac ata_piix crct10dif_pclmul ttm crc32_pclmul drm_kms_helper xen_netfront libata joydev intel_rapl_perf drm pcspkr scsi_mod psmouse i2c_piix4 floppy button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb crc32c_intel xen_blkfront aesni_intel aes_x86_64 crypto_simd cryptd glue_helper evdev serio_raw [last unloaded: tempesta_lib]
[ 1858.030604] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O 4.14.0-tempesta-amd64 #1 Debian 4.14.32-tfw5-1
[ 1858.034679] Hardware name: Xen HVM domU, BIOS 4.8.5-pre 06/22/2018
[ 1858.037514] task: ffffffffae212480 task.stack: ffffffffae200000
[ 1858.040287] RIP: 0010:__parse_connection+0x5a3/0x680 [tempesta_fw]
[ 1858.043109] RSP: 0018:ffff99b64f0038f8 EFLAGS: 00010206
[ 1858.045676] RAX: 000000000000000d RBX: ffff99b5ee19bdba RCX: 0000000000000005
[ 1858.048797] RDX: 0000000000000000 RSI: ffffffffc06b2bd6 RDI: ffff99b5ee19bdb5
[ 1858.052037] RBP: ffff99b5e3b306c0 R08: ffff99b642607900 R09: ffff99b5ee19bd82
[ 1858.055283] R10: ffff99b605164020 R11: 0000000000000005 R12: ffff99b605164020
[ 1858.058551] R13: 000000000000000b R14: 000000000000000b R15: ffff99b5ee19bdb5
[ 1858.061802] FS: 0000000000000000(0000) GS:ffff99b64f000000(0000) knlGS:0000000000000000
[ 1858.065363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1858.068179] CR2: 000055c25825a2e0 CR3: 0000000016c0a004 CR4: 00000000001606f0
[ 1858.071437] Call Trace:
[ 1858.073240] <IRQ>
[ 1858.074875] ? tfw_http_parse_resp+0x4742/0x7950 [tempesta_fw]
[ 1858.077783] ? handle_irq_event_percpu+0x54/0x80
[ 1858.080165] ? __parse_http_date+0x1b10/0x1b10 [tempesta_fw]
[ 1858.082854] ? ss_skb_process+0xae/0x1f0 [tempesta_fw]
[ 1858.085382] ? __evtchn_fifo_handle_events+0x187/0x1b0
[ 1858.087907] ? tfw_http_msg_process+0x137/0xd80 [tempesta_fw]
[ 1858.090645] ? tfw_http_msg_process+0x782/0xd80 [tempesta_fw]
[ 1858.093373] ? irq_exit+0x3c/0xb0
[ 1858.095351] ? xen_evtchn_do_upcall+0x2c/0x40
[ 1858.097647] ? xen_hvm_callback_vector+0x7d/0x90
[ 1858.099998] ? __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[ 1858.102530] ? xennet_start_xmit+0x3f1/0x5b0 [xen_netfront]
[ 1858.105177] ? tfw_connection_recv+0x4e/0x70 [tempesta_fw]
[ 1858.107796] ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[ 1858.110413] ? ss_tcp_process_data+0x1db/0x440 [tempesta_fw]
[ 1858.113108] ? ss_tcp_state_change+0x8a/0x1c0 [tempesta_fw]
[ 1858.115758] ? tcp_fin+0xab/0x170
[ 1858.117737] ? tcp_data_queue+0x5d5/0xc50
[ 1858.119902] ? tcp_rcv_established+0x27c/0x570
[ 1858.122199] ? tcp_v4_inbound_md5_hash+0x62/0x1b0
[ 1858.124552] ? tcp_v4_do_rcv+0x129/0x1d0
[ 1858.126653] ? tcp_v4_rcv+0x947/0xa50
[ 1858.128678] ? ip_local_deliver_finish+0x9a/0x1c0
[ 1858.131010] ? ip_local_deliver+0x6b/0xe0
[ 1858.133136] ? ip_rcv_finish+0x400/0x400
[ 1858.135216] ? ip_rcv+0x289/0x3c0
[ 1858.137095] ? inet_del_offload+0x40/0x40
[ 1858.139142] ? __netif_receive_skb_core+0x84f/0xb30
[ 1858.141425] ? process_backlog+0xa3/0x160
[ 1858.143414] ? process_backlog+0xa3/0x160
[ 1858.145377] ? net_rx_action+0x28e/0x3f0
[ 1858.147276] ? __do_softirq+0x10f/0x2a8
[ 1858.149128] ? irq_exit+0xae/0xb0
[ 1858.150792] ? xen_evtchn_do_upcall+0x2c/0x40
[ 1858.152743] ? xen_hvm_callback_vector+0x7d/0x90
[ 1858.154761] </IRQ>
[ 1858.156034] ? __sched_text_end+0x3/0x3
[ 1858.157819] ? native_safe_halt+0x2/0x10
[ 1858.159609] ? default_idle+0x1a/0xf0
[ 1858.161360] ? do_idle+0x16e/0x1f0
[ 1858.163007] ? cpu_startup_entry+0x6f/0x80
[ 1858.164849] ? start_kernel+0x462/0x482
[ 1858.166613] ? secondary_startup_64+0xa5/0xb0
[ 1858.168526] Code: 45 78 00 00 00 00 48 89 d8 48 c7 85 80 00 00 00 00 00 00 00 48 c7 45 48 e0 d1 66 c0 4c 29 f8 49 39 c5 0f 86 92 fd ff ff 0f b6 03 <0f> 0b 3c 2c 77 27 48 ba 00 02 00 00 01 10 00 00 48 0f a3 c2 0f
[ 1858.175413] ---[ end trace 9182924a523bc2e1 ]---
[ 1859.162433] [tdb] Close table 'client0.tdb'
[ 1859.257935] [tdb] Close table 'cache0.tdb'
[ 1859.281740] [tdb] Close table 'filter0.tdb'
[ 1859.373219] [tdb] Shutdown Tempesta DB
[ 1861.678141] End test: reconf.test_stress_sched_ratio.SchedRatioStaticCustomSg.test_ratio_del_srvs
Unit tests failures:
Functional tests failures: