JavaScript challenge usability

[krizhanovsky]

Upper timeout bound

It seems if a client receives JS challenge and just closes a browser (quite probable) or reply with some significant delay (unlikely since we have large enough timeout), then it remains blocked. The reason for the upper limit is unclear.

The cookie must have Max-Age attribute equal to the upper limit. It's to be discussed should we block cookies after the Max-Age and the upper timeout bound or not. After all the main reason for the JS challenge is to slow bots down... It seems that Max-Age should be equal to our internal configuration sess_lifetime.

Better user experience

~At the moment we send JS challenge on each request. While it was OK with Cookie challenge since a user doesn't see the redirects, it makes user experience significantly worse for JS challenge since all users now see the message about the browser verification. We must send JS challenge only if a users is suspisious, e.g. exceeded an HTTP requests rate soft limit. This is essentialy https://github.com/tempesta-tech/tempesta/issues/598#issuecomment-414756609 , so the task depends on #598.~ Moved to #488

Rework filtration logic

Resolve the comment in tfw_http_sess_check_redir_mark() from https://github.com/tempesta-tech/tempesta/pull/1746 :

we should not block client on the IP layer always. We should deligate the blocking action to the HTTP layer, which can take different blocking actions, e.g. send an HTTP responce in case of block_action attack reply.

Docs

Please update the Wiki https://github.com/tempesta-tech/tempesta/wiki/Sticky-Cookie#javascript-challenge on the task completion. Following topics must be covered: motivation for the upper bound, Max-Age description and advices how to chose the duration correctly, make configuration examples and use cases.

[vankoven]

The upper limit is protection against scripting. Imagine that there is no upper limit and the JS challenge sets only initial delay. Then the simple script below will pass the JS challenge without actually solving the JS challenge:

1. Send request
2. Sleep 10 seconds
3. Send request with cookie set
4. JS challenge is passed!
5. Send tons of requests in this connection

The main goal of the JS challenge is to mitigate asymmetric DDoS: force attacker to use more resources than Tempesta has. If JS Challenge can be bridged without solving JS quiz, then the feature will harm legitimate users without any security improvement.

I was thinking on JS limits more and came to conclusion, that we're setting them in a wrong way. Current limits may be bridged with simple scripting with 100% success rate. The issue is in the upper limit configuration, from #536:

Configuration process must enforce the value to be greater than delay_range + delay_min and show warning if it's less than delay_range + delay_min + 100. Default value is delay_range + delay_min + 1000 (enough ever fro cross atlantic connections and relatively slow hardware and software).

Say we have JS limits: delay_min=1000 delay_range=1000 delay_limit=2100. Note, the limits is much more strict than the default ones, but this doesn't matter. Tempesta sends delay_min and delay_range values in JS challenge code. Since attackers know the Tempesta documentation and code, they can use simple script to bridge the challenge:

1. Send request
2. Sleep exactly delay_min + delay_range milliseconds
3. Send request with cookie set
4. Challenge is passed

This will have 100% success rate, since Tempesta is configured to wait for delay_limit - delay_range - delay_min milliseconds for slow clients.

The issue happen, because allowed time interval to pass the challenge has:

• lower border, randomly generated for each client
• upper border, statically defined for all clients

To solve the issue the upper border must also be generated randomly for each client. Thus we should update delay_limit definition:

delay_limit, an optional argument, specifies maximum difference between current jiffies value when the consequent request is received and a timestamp specified in the sticky cookie minimum time when the consequent request could be received.

Attacker chances to pass the challenge with simple scripting is delay_limit / delay_range * 100%. Seems like values delay_min=1000 delay_range=5000 delay_limit=500 can be reasonable defaults.

[vankoven]

The issue described in https://github.com/tempesta-tech/tempesta/issues/1102#issuecomment-439387230 was resolved, but it has nothing to do with Max-Age header in original issue description.

[krizhanovsky]

During the JSCH testing I observerd annoying flood in dmesg of following records:

[11329907126.692329] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907126.696699] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907126.700651] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907126.705363] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907126.709263] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907126.713075] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907126.716743] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907126.721337] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907126.725148] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907227.014372] net_ratelimit: 16 callbacks suppressed
[11329907227.017500] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907227.024708] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907227.028798] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907227.032784] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907227.037107] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907227.041167] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907227.046295] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907227.051470] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907227.055659] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907227.060512] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907233.470256] net_ratelimit: 15 callbacks suppressed
[11329907233.473285] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907233.479825] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907233.484005] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907233.489294] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1
[11329907233.493202] [tempesta fw] Warning: request dropped: can't send JS challenge: 192.168.100.1

[krizhanovsky]

Sometimes I see that a browser can't load a page fully as on the screenshot

[krizhanovsky]

Just hit the JSCH problem with configuration

sticky {
        cookie name=my_js_cookie enforce max_misses=3 timeout=3;
        js_challenge resp_code=503 delay_min=1000 delay_range=1000 delay_limit=3000 /root/tempesta/etc/js_challenge.html;
}

Accessed a web site protected by Tempesta FW - my browsers accessed it normally as expected. But when I accessed it the next day, my IP was blocked since the browser accessed the site with the same cookie, which was considered by Tempesta as malicious due to expired timestamp inside the cookie.

Setting option options="Max-Age=600" solved the problem. As described in the first comment https://github.com/tempesta-tech/tempesta/issues/1102#issue-381379465 we should set the option equal to the upper bound.

This is a configuration problem and the run script should care about it, not Tempesta FW. The script should check the configuration and print a waning, ideally with link to the wiki https://github.com/tempesta-tech/tempesta/wiki/Sticky-Cookie#session-lifetime

Also update the wiki to say explicitly that Max-Age cookie option is required to correct site operation.

[krizhanovsky]

There is annoying warning

Warning: js_challenge: 'delay_limit' is too big, attacker may hardcode bots and breach the JavaScript challenge with 300% success probability

The problem with the warning is that there is no such 300% probability and it doesn't say how to fix the problem. Please fix the warning to something more meaningful.

[krizhanovsky]

We just received a user claim on failed cookie challenge with configuration without Max-Age.

Also need to deploy the next version of our web site with JSCH to well test it on real client requests.

TBD: how to manage JSCH according to GDPR? In particular how can we provide a user a choice whether to store the cookie and what should we do if we have enforce cookie and a user chooses no to store cookie?

[krizhanovsky]

https://github.com/tempesta-tech/tempesta/pull/1746 has improved the feature usability issue from the user request, so I move the task back to 0.8.

[krizhanovsky]

I think we should just abandon the upper level time limit. As it was commented, bots still can make a sleep within an allowed time range, but the upper bound complicates administration and the code support.

The Tempesta JS challenge must solve the only one task - to slow down the bots. I.e. it's a pure L7 DDoS mitigation.

Modern bots can do much more than random sleep, see for example how ZenRows bypass Cloudflare. This is the task for more sophisticated ML logic and much more advanced JavaScript code, which is scheduled for the enterprise version.

[EvgeniiMekhanik]

After discussion we decide to save upper level limit

[krizhanovsky]

~When the task is done, please reassign it to me to deploy the JSCH on our website to test the feature in the real life~

I created https://github.com/tempesta-tech/tempesta-tech.com/issues/93 for this

[RomanBelozerov]

After discussion:

• [x] ip_block should be moved from the frang to the top level of configurations. Sticky cookie and JS challenge MUST NOT block IP by default. They MUST disconnect the client connection;
• [x] max_misses counter MUST be added to Tempesta and removed from cookie;
• [x] The client connection MUST be closed when number of requests with incorrect cookie (or w/o cookie) is greater than max_misses;
• [x] Tempesta MUST increase max_misses when client does not wait for min_time from JS challenge;
• [x] Tempesta MUST increase max_misses when client send request with old cookie or without cookie;
• [x] max_misses MUST be 1 by default;
• [x] min_time = timestamp (current) + delay_min + timestamp (from cookie) % delay_range
• [x] Tempesta MUST return a 400 response and FIN TCP when block_action attack reply is present. Now I receive 503 (w/o cookie) + FIN TCP;
• [x] reworking the calculation on Tempesta's side. Tempesta MUST restart JS challenge if the current timestamp is less than timestamp from cookie.
• [x] cookie enforce MUST be present for JS challenge. max_misses works only with enforce.

[EvgeniiMekhanik]

Closed by https://github.com/tempesta-tech/tempesta/pull/2025