Closed krizhanovsky closed 4 years ago
I also noticed max_misses
misbehaviour while testing the issue. js_challenge
directive was removed from configuration above. Chome initially connects to http://192.168.122.12/some_uri
and receives redirect to http://192.168.122.12/jdkey=000000010000000103fbbecb5ebf05fd3610ba349be9723c06755e03db976195/some_uri
. But then instead of stripping redirect tag, the request is passed to origin, and Chrome gets 404 error from origin server. No errors appear in Tempesta log, but browser actually passed the challenge, next request in the same browser session gives the site content.
There is definitely relation to #1102 . But this doesn't related to instant failure of JS challenge, it doesn't allow user to reconnect if he has an old sticky cookie value and no session stored in Tempesta databases. Need to rerun JS challenge after some time, e.g. after delay_min + delay_range + ..
.
js_challenge.html.txt The redirection file from my test
The redirection file from my test
Yeah, this file from some very-very old installation, it had issues with reading cookie value from response, thus no delay was done between reties and Tempesta blocked client. I also experienced absolutely the same issue (: Took me some time to discover that.
The issue was faced because we erroneously treated all requests with header Accept: */*
header as supporting cookies. That was the biggest mistake I've done on JS challenge implementation, trying to support challenge for any link typed into browser's search bar.
Actually browsers sends Accept: */*
in every request, so all requests was treated as challengeable, which can lead to false challenge fails. As client follows some link or manually enters a line into search bar, the very first request is used to load DOM. This request always has Accept: text/html
and is the right target for JS challenge. In the same time. Even if the uri is direct link to some image, thus any first request to the protected site will imply Accept: text/html
header.
The issue I saw with JS challenge stability in some cases was because of unlucky coincidence related connection close on second request (request for favicon.ico
). After Accept
issue was fixed, JS challenge started to work reliable.
Just tried a clean setup from with current #1395. Deployed our website under Apache HTTPD. Sent several requests with Firefow and Chrome, one after another. There were warnings about bad cookies and ratelimits, just wait a little bit and press Refresh in a browser. Got this crash (didn't check this on master):
[ 137.693007] [tempesta fw] Warning: http_sess: bad received HMAC value for 192.168.100.1: 7(pos=0), ts=0xffffc14e orig_hmac=[a33793fd52f9bc1d5786ca2f27fc762790f3c4ee]
[ 140.000413] [tempesta fw] Warning: request dropped: sticky cookie challenge was failed: 192.168.100.1
[ 160.318895] [tempesta fw] Warning: frang: connections max num. exceeded for 192.168.100.1: 9 (lim=8)
[ 161.777545] [tempesta fw] Warning: frang: connections max num. exceeded for 192.168.100.1: 9 (lim=8)
[ 161.784842] [tempesta fw] Warning: frang: connections max num. exceeded for 192.168.100.1: 9 (lim=8)
[ 161.792483] [tempesta fw] Warning: frang: connections max num. exceeded for 192.168.100.1: 9 (lim=8)
[ 161.796213] [tempesta fw] Warning: frang: connections max num. exceeded for 192.168.100.1: 9 (lim=8)
[ 161.855332] general protection fault: 0000 [#1] SMP PTI
[ 161.857284] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) binfmt_misc crct10dif_pclmul bochs_drm ttm drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect ppdev sysimgblt sg serio_raw parport_pc parport button ip_tables x_tables ext4 crc16 mbcache jbd2 fscrypto sd_mod sr_mod cdrom ata_generic ata_piix libata e1000 scsi_mod i2c_piix4 psmouse
[ 161.868959] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 4.14.32-kdump+ #147
[ 161.872214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20190727_073836-buildvm-ppc64le-16.ppc.fedoraproject.org-3.fc31 04/01/2014
[ 161.877325] task: ffff9de6663f50c0 task.stack: ffff9de666854000
[ 161.880072] RIP: 0010:__memcpy_fast+0x148/0x160 [tempesta_lib]
[ 161.882486] RSP: 0018:ffff9de67fd03878 EFLAGS: 00010286
[ 161.885785] RAX: 00013da600000002 RBX: 0000000000000000 RCX: ffffffffc05648b4
[ 161.888860] RDX: 0000000000000002 RSI: 0000000000000a0d RDI: 00013da600000000
[ 161.890536] RBP: ffffffffc057bc40 R08: ffffffffc05648b2 R09: ffff9de666d20cc0
[ 161.892585] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[ 161.894708] R13: 0000000000001000 R14: 0000000000000002 R15: ffff9de666d39188
[ 161.897048] FS: 0000000000000000(0000) GS:ffff9de67fd00000(0000) knlGS:0000000000000000
[ 161.899298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 161.902892] CR2: 00007fdb59ff9fe8 CR3: 000000006520a001 CR4: 00000000003606e0
[ 161.905441] Call Trace:
[ 161.907198] <IRQ>
[ 161.908975] tfw_http_msg_expand_data+0x11b/0x370 [tempesta_fw]
[ 161.911502] ? tfw_http_conn_drop+0x250/0x250 [tempesta_fw]
[ 161.913725] tfw_cache_do_action+0xd43/0xdf0 [tempesta_fw]
[ 161.916541] ? tfw_hash_str_len+0x147/0x160 [tempesta_fw]
[ 161.918822] ? tfw_http_sticky_add+0x270/0x270 [tempesta_fw]
[ 161.921319] ? tfw_http_conn_drop+0x250/0x250 [tempesta_fw]
[ 161.922971] ? tfw_http_req_key_calc+0xc5/0xe0 [tempesta_fw]
[ 161.924387] ? tfw_http_conn_drop+0x250/0x250 [tempesta_fw]
[ 161.925773] tfw_cache_process+0xb7/0x290 [tempesta_fw]
[ 161.927703] ? tfw_gfsm_move+0x146/0x180 [tempesta_fw]
[ 161.928982] tfw_http_msg_process_generic+0x745/0xcf0 [tempesta_fw]
[ 161.930365] ? mod_timer+0x16c/0x3a0
[ 161.931499] tfw_http_msg_process+0x7f/0xc0 [tempesta_fw]
[ 161.934569] __gfsm_fsm_exec+0x51/0x90 [tempesta_fw]
[ 161.936375] tfw_connection_recv+0x3c/0x60 [tempesta_fw]
[ 161.938471] ? tfw_connection_send+0x20/0x20 [tempesta_fw]
[ 161.939892] ss_tcp_process_data+0x1e5/0x480 [tempesta_fw]
[ 161.941169] ? mod_timer+0x16c/0x3a0
[ 161.942230] ss_tcp_data_ready+0x3e/0x90 [tempesta_fw]
[ 161.943485] tcp_rcv_established+0x4cd/0x570
[ 161.944669] tcp_v4_do_rcv+0x124/0x1c0
[ 161.945722] tcp_v4_rcv+0x93f/0xa40
[ 161.946722] ip_local_deliver_finish+0x95/0x1c0
[ 161.947911] ip_local_deliver+0x66/0xe0
[ 161.949649] ? tcp_v4_early_demux+0x10d/0x140
[ 161.951257] ? ip_rcv_finish+0x175/0x400
[ 161.952886] ip_rcv+0x284/0x3b0
[ 161.953894] ? inet_del_offload+0x30/0x30
[ 161.954993] __netif_receive_skb_core+0x84a/0xb30
[ 161.956091] ? process_backlog+0x92/0x120
[ 161.957160] process_backlog+0x92/0x120
[ 161.958130] net_rx_action+0x261/0x3a0
[ 161.959103] __do_softirq+0x104/0x297
[ 161.960079] irq_exit+0xa3/0xb0
[ 161.961007] do_IRQ+0x45/0xc0
[ 161.961844] common_interrupt+0x7a/0x7a
[ 161.963109] </IRQ>
[ 161.964560] RIP: 0010:default_idle+0x12/0xf0
[ 161.965914] RSP: 0018:ffff9de666857ec0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffc4
[ 161.967915] RAX: ffffffffa8e28510 RBX: ffff9de6663f50c0 RCX: 0000000000000000
[ 161.970846] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 161.972246] RBP: 0000000000000001 R08: 0000000000000002 R09: 0000000000000001
[ 161.973648] R10: 00000000ffffc9eb R11: 0000000000000000 R12: ffff9de6663f50c0
[ 161.975005] R13: ffff9de6663f50c0 R14: 0000000000000000 R15: 0000000000000000
[ 161.976282] ? __sched_text_end+0x3/0x3
[ 161.977184] do_idle+0x156/0x1b0
[ 161.978045] cpu_startup_entry+0x6a/0x70
[ 161.978948] start_secondary+0x188/0x1b0
[ 161.979883] secondary_startup_64+0xa5/0xb0
[ 161.980819] Code: f8 f6 c2 04 0f 84 25 ff ff ff 8b 31 48 83 c0 04 48 83 c1 04 89 70 fc f6 c2 02 0f 84 18 ff ff ff 0f b7 31 48 83 c0 02 48 83 c1 02 <66> 89 70 fe 83 e2 01 74 05 0f b6 11 88 10 c3 66 0f 1f 84 00 00
[ 161.985038] RIP: __memcpy_fast+0x148/0x160 [tempesta_lib] RSP: ffff9de67fd03878
[ 161.986871] ---[ end trace e55639d14da07e8e ]---
[ 161.987974] Kernel panic - not syncing: Fatal exception in interrupt
[ 161.989417] Kernel Offset: 0x27800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 161.991164] Rebooting in 60 seconds..
I couldn't reproduce the crash from the message above.
I also noticed max_misses misbehaviour ....
This was detached into #1397 #1398
Created separate https://github.com/tempesta-tech/tempesta/issues/1399 for the crash - the bug is hard to reproduce and could be unrelated to this task.
Scope
Tempesta FW with enabled JS challenge can not service a request from a browser. There was an external report for 0.6.2 release about the JS challenge failure with
Warning: request dropped: sticky cookie challenge was failed
. I tested the config as the below:on current Tempesta FW master running in a VM with Apache HTTPD on loopback serving simple 2 byte
index.html
. I used Firefox browser on the host system to request Tempesta FW. I cleared cookies in the browser before the try.I'm attaching the log of tcpdump tcpdump.txt ran on the host system. It seems the reason is that the browswer requests
favicon.ico
right after we sent the challenge, so Tempesta FW treats the request as an attack and blocks the browser.Maybe related, but I'm not sure, with https://github.com/tempesta-tech/tempesta/issues/1102
The Tempesta FW log with
DEBUG=3
:Testing
Also please