rate limiting

[tarekziade]

Let's implement this rule:

{ 
"request_rate": "1 / 60 / IP",
}

where:

• 1 is the number of requests of the window
• 60 the duration in seconds of the window
• IP the method to distinguish clients. (can be IP or FF (forwarded-for) )

Extra headers can be used to uniquely identify a client - they are separated by +.

For example, if your web services uses browser id assertions, you can do:

{ 
"request_rate": "1 / 60 / IP+Authorization+User-Agent",
}

Notice that since DDos attacks will likely not use valid authorization headers, this
rate limiting has to be used for limiting legit clients.

DDos attacks needs to be prevented using only the remote address

[tarekziade]

@jvehent how does that look ?

[jvehent]

It looks good. I'm wondering if the stringified format will be flexible enough though. What if we decompose it?

{
    "limits": {
        "rate": {
            "seconds": 60,
            "hits": 10,
            "match": "header:Authorization AND header:User-Agent"
        },
        "rate": {
            "seconds": 10,
            "hits": 100,
            "match": "header:X-Forwarded-For OR ipv4 OR ipv6"
        }
    }
}

The limits above would allow a client identified by its Authorization header to send 10 requests within 60 seconds. Another limit would max up the number of requests coming from a single IP to 100 in 10 seconds.

If we use the AND operator to handle multi-criteria filters, lua will need a way to hash the criterias together and store that as a key in the rate limiting table. Is it realistic speed-wise?

[tarekziade]

Your syntax looks good.

If we use the AND operator to handle multi-criteria filters, lua will need a way to hash the criterias together and store that as a key in the rate limiting table. Is it realistic speed-wise?

As long as the rules are easy to convert into Lua, I can create a Lua function and compile it on the fly then use it. So no problem

[jvehent]

Cool, then let's try that. A minor change, since my json above used the rate key twice, rates would be better as an array of rate limits.

{
    "limits": {
        "rates": [
            {
                "seconds": 60,
                 "hits": 10,
                "match": "header:Authorization AND header:User-Agent"
            },
            {
                "seconds": 10,
                "hits": 100,
                "match": "header:X-Forwarded-For OR ipv4 OR ipv6"
            }
        ]
    }
}

[mpurzynski]

Let's have an expiration period per rule. Also, the period should be randomized - if I say expire in 30 minutes, it can be randomized from 30-N to 30+N to make attacker's life harder for discovering 'how long does it take for them to unblock me'. The N might be another field here.

[jvehent]

@mpurzynski : I think what you are referering to is a blacklist expiration. The rate limit is designed to be a sliding window. It limits the rate of requests within the window of X seconds, such that an additional expiration rule is not needed.

[tarekziade]

Please let's add parenthesis when using AND, that makes my tokenization work so much easier :)

For example:

header:Authorization AND header:User-Agent AND (header:X-Forwarded-For OR ipv4 OR ipv6)

or

(header:X-Forwarded-For OR ipv4 OR ipv6)

[tarekziade]

work in progress at https://github.com/mozilla/videur/pull/10

[tarekziade]

one question : what "ipv4 OR ipv6" is supposed to do exactly ?