Block users abusing the server

[vkuznet]

We need a general type of protection against users who accidentally create DDoS behavior. It can be done at front-end layer, who can count number of user requests and block specific users if such behavior would be observed.

In addition we can create lemon alarm to spot this behavior.

[ghost]

lat: At the moment the front-end can do very little for DAS as it doesn't require authentication, and distinguishing users by IP address is definitely not going to be something I'd find appealing / acceptable. It's just too coarse a gun.

If DAS becomes authenticated, there are mechanisms to block access by certificate/login; that mechanism has already been in place for 1+ year. We'd need to very carefully define how multi-front-end service will detect overload, and feed back into the blacklist, and how things expire from the blacklist.

Also we need to be careful to make any rate limits or other restrictions service specific, as there are other services which can easily sustain high rate of requests without getting into trouble. Hence I would also suggest we need to keep open the option that DAS backend itself needs to monitor access rate and spot anomalous behaviour, and start responding with some specific HTTP error code when the limits are exceeded. It would, for example, be easier to detect a storm of 'bogus' requests in DAS back-end, than try to detect it in the front-end.

[ghost]

lat: Note also that a perfectly reasonable alternative in DAS backend is to detect invalid requests - those that say fail input validation - and artificially slow them down, either always or after it has detected certain high rate of recent accesses.

For situations like the serial client recently that will automatically fix the problem. It will also partially solve the issue if the accesses come from a small number of hosts, whether serial or parallel. It will not solve DDoS situation where the accesses come from numerous hosts in parallel, as then it will just tie up all the DAS server threads, and other (valid) requests will time out.

[vkuznet]

valya: I'm not sure I understand how server can influence a client from sending bunch of requests and slowing them down, please explain. If request is invalid DAS throws HTTP error.

[ghost]

lat: Add time.sleep(0.5) in exception catch handler when you detect excessive error rate. For the more general version, search google for 'tarpit' and 'honeypot'. Various intrusion detection systems do things like responding one byte at a time every say 5 seconds when they detect they are under attack, which ties up any dumb clients as they (hopelessly) try to read back responses.

[vkuznet]

valya: Thanks, will read about it.

My misunderstanding came from conclusion that client can send as many requests as it wants. The timeout will only slow down client ability to read, but can't influence its ability to send.

[ghost]

lat: Correct, hence the comment about serial clients vs. parallel ones.

[vkuznet]

valya: I added new onhold queue which will slow down users who will abuse DAS server with sequential requests. This is done in the following way:

• I add monitoring into DAS cache method used by non-web clients
• If a user (based on its IP) places too many queries (determined by daily usage), DAS server will trigger them and place requests above threshold into onhold queue.
• Every record in onhold queue has timestamp which will be used by processing thread to determine when it should do a job
• user will get onhold status for requests which were put aside into onhold queue.

Changes went into 4f6abbe572812a4c3fe5a9d8918771974b772258

[ghost]

lat: Sounds good Valentin, thanks! One small comment, 'print "\n### get_nhits", nhits' is ok for now but let's make sure it gets turned off eventually.

[vkuznet]

valya: Ahh, thanks, that's oversight on my side, certainly it will be removed.