search

nlopes / actix-web-prom

Actix-web middleware to expose Prometheus metrics
MIT License
89 stars 56 forks source link

Should not extend metrics response on 404 (Uncontrolled Resource Consumption Vulnerability) #80

Open oscarhermoso opened 6 months ago

oscarhermoso commented 6 months ago

Currently, when an endpoint receives a 404, it extends the Prometheus metrics response by approx ~15 new lines / ~1.5KB.

For example, my server does not include a /time.php endpont, so when I was bot scraped by a bot, of these lines were appended.

# ...
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.005"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.01"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.025"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.05"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.1"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.25"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.5"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="1"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="2.5"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="5"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="10"} 1
public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="+Inf"} 1
public_api_http_requests_duration_seconds_sum{endpoint="/time.php",method="GET",status="404"} 0.000011828
public_api_http_requests_duration_seconds_count{endpoint="/time.php",method="GET",status="404"} 1
# ...
public_api_http_requests_total{endpoint="/time.php",method="GET",status="404"} 1

Then, repeat for 100+ other 404'ing endpoints, and you end up with a really large metrics response, resulting in hundreds of GB of egress on your /metrics endpoint (which has been kindly refunded by the Fly.io billing team).

Screenshot from 2024-01-26 15-03-36

Furthermore, a dedicated attacker could certainly exhaust significantly more resources simply by 404ing on more URLs

To resolve this issue, 404 responses from endpoints that fail to match should not be included in metrics. I believe this PR should resolve the issue?

Also recommend submitting an advisory to RustSec for this version of the package.

oscarhermoso commented 6 months ago

Just found https://github.com/nlopes/actix-web-prom/pull/71 and exclude_status(actix_http::StatusCode::NOT_FOUND)

I have applied this temporary work around to my repo, but is not in the docs or readme