Add rate limiting to authentication endpoints

garethbowen commented 4 years ago

Currently it's possible for a malicious actor to make unlimited attempts at logging in until correct credentials are found. This sort of brute force attack should be limited by rate limiting failed authentication requests. This goes for both logging in through the login form, as well as basic authentication via a range of URLs.

Denial of service style attacks are out of scope for this issue.

MaxDiz commented 4 years ago

bumping to 3.13 based on discussion with @garethbowen

craig-landry commented 3 years ago

I've asked @latin-panda to look into this one.

latin-panda commented 3 years ago

Hi @garethbowen I noticed this got removed from "ready for dev", is it because of arch v3? or maybe the plans changed?

I have some questions regarding expectations of what we want to achieve, this is something we can build ourselves or use a library. Today I took a look at the libraries and it really depends on what we want to do with this feature.

There are many, to mention some:

node-rate-limiter-flexible: I like this library because it offers several features for more complex solutions like add/reduce quota of rate limit for certain users, it doesn't need plugin to work with Redis or PostgreSQL, allows to control requests rate in single process or distributed environment, Typescript declaration supported (maybe someday for api), set headers in case we want to expose that info:
```
const headers = {
"Retry-After": rateLimiterRes.msBeforeNext / 1000,
"X-RateLimit-Limit": opts.points,
"X-RateLimit-Remaining": rateLimiterRes.remainingPoints,
"X-RateLimit-Reset": new Date(Date.now() + rateLimiterRes.msBeforeNext)
}
```
express-rate-limit: Light, no dependencies, uses the time window approach, uses in-memory Node process, it can use Redis but with a plugin. This one is nice if we're looking to implement something super simple like: reject requests if exceeds 100 requests in 1 min. Very fast to implement, just plug it to express as middleware and done.
```
const limiter = rateLimit({
windowMs: 15 * 60 * 1000, // 15 minutes
max: 100 // limit each IP to 100 requests per windowMs
});
```

// apply to all requests app.use(limiter);


- [express-slow-down](https://www.npmjs.com/package/express-slow-down): Very basic implementation of rate limit that doesn't block but slow downs responses.
- [express-brute](https://www.npmjs.com/package/express-brute) [Discarded] Not updated in 5 years.
- [express-limiter](https://www.npmjs.com/package/express-limiter) [Discarded] Not updated in 4 years.

I think if we just need a basic solution then `express-rate-limit` is enough, for more complexity then `node-rate-limiter-flexible`. 
Do we want to save stats in telemetry?
Do we want to block the request if they reach the limit? 
Is 100~ per minute what we're looking for? Or perhaps windows of 1h? 
Is the quota (limit) flat for everybody? Or perhaps Admin users have more privilege?

We can also do some research to determine these answers, then make some performance tests by setting this according with the expectations.

garethbowen commented 3 years ago

I think this got removed from 3.13 when we changed from projects to milestones, so don't read too much into that!

There's some discussion in the private issue about how to go about this. As well as looking into JS libraries we should look at our cloud hosting providers to see if they have something that will just work. We would then need to document how to set it up for our self-hosting partners. Reach out to SRE to discuss options here.

My main concern is about an attacker brute forcing credentials, rather than DOSing the server, so focus on the login and basic auth pipelines and we should be good.

latin-panda commented 3 years ago

I've continued the discussion in the other ticket mentioned above.

garethbowen commented 1 year ago

@ralfudx This is the rate limiting PR I was talking about in today's call.

The idea is to stop attackers being able to have unlimited tries at finding a correct username and password.

Basically the idea is that it should block any attacker that is...

trying a bunch of passwords for the same user,
trying a bunch of users with the same password, or
trying a bunch of combinations from the same IP.

I've set it to allow up to 10 tries every 10 seconds so I hoping actual users will never hit the limit. I'm only blocking authentication requests, so you can still access things like static resources after you've been blocked, but you can't log in until the 10 seconds is up. The CHT supports logging in via the login page (form submitted to the login api), AND basic auth and it's important that both are protected.

This change does not protect against denial of service type attacks where someone just overloads the system - that's out of scope for now.

Any help with QA to ensure that all attack routes are blocked, and no real users are blocked, would be much appreciated!

ralfudx commented 1 year ago

@garethbowen I'm still having having issues with my local env set-up... I've reached out a couple engineers for this (@njuguna-n @Benmuiruri ) once this is resolved i'll proceed with testing this...

garethbowen commented 1 year ago

@ralfudx For testing this I would recommend using the docker helper to install the 6530-rate-limiting docker image. This way you don't have to set up the whole dev environment (though this may be useful later). Instructions here may help: https://docs.communityhealthtoolkit.org/apps/guides/hosting/4.x/app-developer/#cht-docker-helper-for-4x

ralfudx commented 1 year ago

@garethbowen i was finally able to test this after setting-up my cht-core dev env (thanks to @Benmuiruri & @njuguna-n) I noticed one of the AC's is not been fulfilled trying a bunch of passwords for the same user ✅ - returned a 429 status code after 10 failed tries trying a bunch of users with the same password ✅ - returned a 429 status code after 10 failed tries trying a bunch of combinations from the same IP ❌ I was able to continuously send a combination of different creds (user & password) and got a 401 each time

garethbowen commented 1 year ago

@ralfudx Thanks for the quick turnaround!

Just checking but that screenshot shows a 429 at the end - was that taken from one of the tests that passed or the IP address check that failed?

Can you please provide the logs from the test that failed?

Also could you provide the steps you followed to test? For example, was this from the login page? If so, how did you manage to change the username and password for each request and still manage 10 requests in 10 seconds?

ralfudx commented 1 year ago

Hi @garethbowen yes the screenshot is from one of the tests that passed (trying a bunch of passwords for the same user) I tested this using postman directly hitting the login endpoint (http://localhost:5988/medic/login) with a post request Steps

Create post request with this payload: { "user": "{{user}}", "password": "{{password}}", "redirect": "http://localhost:5988/#/messages", "locale": "en" }
Create Pre-request Script with "user" and "password" global variables const user = Math.floor((Math.random()*100 +1)); pm.globals.set("user", "medic"+user);

const password = Math.floor((Math.random()*100 +1)); pm.globals.set("password", "password"+password);

For the third test (that Failed) - just send the request multiple times within 10 seconds
For the first test - replace "{{user}}" with "medic" - then comment out the variable for "user" in the Pre-request Script - then send the request multiple times within 10 seconds (you should get a 429 on the 10th request)
For the second test - replace "{{password}}" with "password" - then comment out the variable for "password" in the Pre-request Script - then send the request multiple times within 10 seconds (you should get a 429 on the 10th request)

ralfudx commented 1 year ago

@garethbowen here's the log from the failed test

2023-11-09T16:36:16.682 REQ: c4ee7849-7eb3-473f-a7f0-cec10cd26a12 ::1 - GET /medic/_design/medic-client/_view/data_records_by_type?group=true HTTP/1.1
2023-11-09T16:36:16.685 REQ: dfd74539-01dc-41db-93b8-8f4b55bdadea ::1 - GET /medic-user-medic-meta/_design/medic-user/_view/read?group=true HTTP/1.1
2023-11-09T16:36:16.695 RES: c4ee7849-7eb3-473f-a7f0-cec10cd26a12 ::1 - GET /medic/_design/medic-client/_view/data_records_by_type?group=true HTTP/1.1 200 - 12.300 ms
2023-11-09T16:36:16.699 RES: dfd74539-01dc-41db-93b8-8f4b55bdadea ::1 - GET /medic-user-medic-meta/_design/medic-user/_view/read?group=true HTTP/1.1 200 - 13.587 ms
2023-11-09T16:36:30.785 DEBUG: Checking for a configured outgoing message service 
2023-11-09T16:36:31.003 REQ: e52af369-f913-45bd-8e62-8c9811bd5cc1 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:31.015 RES: e52af369-f913-45bd-8e62-8c9811bd5cc1 ::1 - POST /medic/login HTTP/1.1 401 25 11.345 ms
2023-11-09T16:36:31.526 REQ: 46113363-01a3-4eef-847e-d6bc6385c0c0 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:31.540 RES: 46113363-01a3-4eef-847e-d6bc6385c0c0 ::1 - POST /medic/login HTTP/1.1 401 25 13.173 ms
2023-11-09T16:36:32.042 REQ: 4758c0b1-8e92-415c-8c62-41c3e932a2b7 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:32.050 RES: 4758c0b1-8e92-415c-8c62-41c3e932a2b7 ::1 - POST /medic/login HTTP/1.1 401 25 8.015 ms
2023-11-09T16:36:32.572 REQ: 09dc44bf-3ebd-4d08-a5f5-f7b746eb7822 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:32.583 RES: 09dc44bf-3ebd-4d08-a5f5-f7b746eb7822 ::1 - POST /medic/login HTTP/1.1 401 25 10.270 ms
2023-11-09T16:36:33.225 REQ: 36ef3297-eeaa-4ece-a758-0fa8ceb22552 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:33.236 RES: 36ef3297-eeaa-4ece-a758-0fa8ceb22552 ::1 - POST /medic/login HTTP/1.1 401 25 10.189 ms
2023-11-09T16:36:33.767 REQ: 2c28f797-19ca-43e3-9a17-41c23b829010 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:33.780 RES: 2c28f797-19ca-43e3-9a17-41c23b829010 ::1 - POST /medic/login HTTP/1.1 401 25 12.422 ms
2023-11-09T16:36:34.303 REQ: 0f0562d6-b2b4-4b2b-aa15-1def428e9c80 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:34.317 RES: 0f0562d6-b2b4-4b2b-aa15-1def428e9c80 ::1 - POST /medic/login HTTP/1.1 401 25 12.579 ms
2023-11-09T16:36:34.877 REQ: 358edf7a-027a-4902-9673-6fe1dec99b59 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:34.889 RES: 358edf7a-027a-4902-9673-6fe1dec99b59 ::1 - POST /medic/login HTTP/1.1 401 25 11.315 ms
2023-11-09T16:36:35.427 REQ: a89c9880-aecc-4fb7-9260-61b3d3e15311 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:35.438 RES: a89c9880-aecc-4fb7-9260-61b3d3e15311 ::1 - POST /medic/login HTTP/1.1 401 25 10.070 ms
2023-11-09T16:36:35.967 REQ: 71b86e89-5f72-4553-aa51-88d9c190145c ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:35.982 RES: 71b86e89-5f72-4553-aa51-88d9c190145c ::1 - POST /medic/login HTTP/1.1 401 25 15.119 ms
2023-11-09T16:36:36.603 REQ: d01a6f30-7c1c-463c-87af-8e80c91c19d7 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:36.608 RES: d01a6f30-7c1c-463c-87af-8e80c91c19d7 ::1 - POST /medic/login HTTP/1.1 401 25 5.227 ms
2023-11-09T16:36:37.431 REQ: c73f315c-03ed-45ab-ba39-213da52bad17 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:37.442 RES: c73f315c-03ed-45ab-ba39-213da52bad17 ::1 - POST /medic/login HTTP/1.1 401 25 10.338 ms
2023-11-09T16:36:38.054 REQ: 957e3cac-744c-4fa6-ada4-e2a4c379323c ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:38.068 RES: 957e3cac-744c-4fa6-ada4-e2a4c379323c ::1 - POST /medic/login HTTP/1.1 401 25 13.822 ms
2023-11-09T16:36:38.657 REQ: beb2626e-4e17-48d3-b5ca-d3bf068b75a3 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:38.670 RES: beb2626e-4e17-48d3-b5ca-d3bf068b75a3 ::1 - POST /medic/login HTTP/1.1 401 25 12.050 ms
2023-11-09T16:36:39.286 REQ: 634ac5bb-6001-4875-b1c5-6e3ea6ed2091 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:39.299 RES: 634ac5bb-6001-4875-b1c5-6e3ea6ed2091 ::1 - POST /medic/login HTTP/1.1 401 25 12.489 ms
2023-11-09T16:36:40.010 REQ: b1015c7b-ee7a-413c-a501-c2b83bf9a68e ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:40.024 RES: b1015c7b-ee7a-413c-a501-c2b83bf9a68e ::1 - POST /medic/login HTTP/1.1 401 25 12.933 ms
2023-11-09T16:36:40.730 REQ: 37379be5-b378-4bf7-ae33-0939952c41a5 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:40.737 RES: 37379be5-b378-4bf7-ae33-0939952c41a5 ::1 - POST /medic/login HTTP/1.1 401 25 7.263 ms
2023-11-09T16:36:41.458 REQ: a8717e70-70dc-4e90-9197-638c70062e30 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:41.469 RES: a8717e70-70dc-4e90-9197-638c70062e30 ::1 - POST /medic/login HTTP/1.1 401 25 9.800 ms
2023-11-09T16:36:42.076 REQ: bfc605e3-5da2-42fd-98b9-6aa3ca593134 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:42.096 RES: bfc605e3-5da2-42fd-98b9-6aa3ca593134 ::1 - POST /medic/login HTTP/1.1 401 25 18.520 ms
2023-11-09T16:36:42.750 REQ: 5f4f3709-6cbe-4130-97a9-a1b23e3c42bb ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:42.766 RES: 5f4f3709-6cbe-4130-97a9-a1b23e3c42bb ::1 - POST /medic/login HTTP/1.1 401 25 15.703 ms
2023-11-09T16:36:43.465 REQ: 096aa5dd-16cb-4b20-a49f-c6910ba1488f ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:43.473 RES: 096aa5dd-16cb-4b20-a49f-c6910ba1488f ::1 - POST /medic/login HTTP/1.1 401 25 6.866 ms
2023-11-09T16:36:44.149 REQ: 65f635dc-3e51-4167-8af3-5c4913c5c076 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:44.165 RES: 65f635dc-3e51-4167-8af3-5c4913c5c076 ::1 - POST /medic/login HTTP/1.1 401 25 15.558 ms
2023-11-09T16:36:44.705 REQ: 803b7be9-5f20-48e3-a047-d155225e0ca3 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:44.724 RES: 803b7be9-5f20-48e3-a047-d155225e0ca3 ::1 - POST /medic/login HTTP/1.1 401 25 17.805 ms
2023-11-09T16:36:45.282 REQ: 39e842db-5fd5-4f86-b882-4558971d98ed ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:45.295 RES: 39e842db-5fd5-4f86-b882-4558971d98ed ::1 - POST /medic/login HTTP/1.1 401 25 12.373 ms
2023-11-09T16:36:45.846 REQ: 9c8593ad-359f-4973-b5c0-cfaa5560f2c0 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:45.858 RES: 9c8593ad-359f-4973-b5c0-cfaa5560f2c0 ::1 - POST /medic/login HTTP/1.1 401 25 9.146 ms
2023-11-09T16:36:46.408 REQ: 0e587dc0-00d0-4736-a428-a76edd570640 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:46.419 RES: 0e587dc0-00d0-4736-a428-a76edd570640 ::1 - POST /medic/login HTTP/1.1 401 25 10.488 ms
2023-11-09T16:36:46.966 REQ: 56423b05-41d5-44b8-8c39-9e801a0fcccc ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:46.982 RES: 56423b05-41d5-44b8-8c39-9e801a0fcccc ::1 - POST /medic/login HTTP/1.1 401 25 15.413 ms
2023-11-09T16:36:47.504 REQ: 3683f87b-a702-450e-994e-95e6126f0475 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:47.525 RES: 3683f87b-a702-450e-994e-95e6126f0475 ::1 - POST /medic/login HTTP/1.1 401 25 18.112 ms
2023-11-09T16:36:48.095 REQ: 4f2fca11-ff6d-40d1-b436-1f47a7cd0add ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:48.106 RES: 4f2fca11-ff6d-40d1-b436-1f47a7cd0add ::1 - POST /medic/login HTTP/1.1 401 25 8.106 ms
2023-11-09T16:36:48.673 REQ: e6e95d67-ed67-4a12-bfda-9eadd07639f8 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:48.683 RES: e6e95d67-ed67-4a12-bfda-9eadd07639f8 ::1 - POST /medic/login HTTP/1.1 401 25 9.943 ms
2023-11-09T16:36:49.205 REQ: 8d6623bb-d571-4d80-845c-e6914ca7498b ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:49.217 RES: 8d6623bb-d571-4d80-845c-e6914ca7498b ::1 - POST /medic/login HTTP/1.1 401 25 11.257 ms
2023-11-09T16:36:49.893 REQ: 7148ee44-bbd0-4531-97cd-2069547e1fee ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:49.907 RES: 7148ee44-bbd0-4531-97cd-2069547e1fee ::1 - POST /medic/login HTTP/1.1 401 25 13.378 ms
2023-11-09T16:36:50.487 REQ: 3bbf7da8-bfae-46de-9e35-445d4efa0398 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:50.500 RES: 3bbf7da8-bfae-46de-9e35-445d4efa0398 ::1 - POST /medic/login HTTP/1.1 401 25 12.301 ms
2023-11-09T16:36:51.058 REQ: 82783e93-0942-4a86-b669-a7eae9e5ddfc ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:51.068 RES: 82783e93-0942-4a86-b669-a7eae9e5ddfc ::1 - POST /medic/login HTTP/1.1 401 25 9.957 ms
2023-11-09T16:36:51.638 REQ: dce0c7cf-ccce-46f6-af3f-cd1b6b82a2e8 ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:51.653 RES: dce0c7cf-ccce-46f6-af3f-cd1b6b82a2e8 ::1 - POST /medic/login HTTP/1.1 401 25 13.586 ms
2023-11-09T16:36:52.203 REQ: dfd51dbb-150b-42b8-a577-5869fb99fbaa ::1 - POST /medic/login HTTP/1.1
2023-11-09T16:36:52.213 RES: dfd51dbb-150b-42b8-a577-5869fb99fbaa ::1 - POST /medic/login HTTP/1.1 401 25 9.502 ms

garethbowen commented 1 year ago

@ralfudx Excellent find! You're absolutely right.

I've now fixed that issue, cleaned up the code a lot, and rebased to get the latest test improvements. Would you mind running your testing again on the latest version?

ralfudx commented 12 months ago

Hi @garethbowen I can confirm that this now works as expected: trying a bunch of passwords for the same user ✅ - returned a 429 status code after 10 failed tries trying a bunch of users with the same password ✅ - returned a 429 status code after 10 failed tries trying a bunch of combinations from the same IP ✅ - returned a 429 status code after 10 failed tries here's a screenshot for the test that failed previously

garethbowen commented 12 months ago

Thanks @ralfudx ! Merged.

medic / cht-core

Add rate limiting to authentication endpoints #6530