Open oscarhermoso opened 6 months ago
Currently, when an endpoint receives a 404, it extends the Prometheus metrics response by approx ~15 new lines / ~1.5KB.
For example, my server does not include a /time.php endpont, so when I was bot scraped by a bot, of these lines were appended.
/time.php
# ... public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.005"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.01"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.025"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.05"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.1"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.25"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="0.5"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="1"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="2.5"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="5"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="10"} 1 public_api_http_requests_duration_seconds_bucket{endpoint="/time.php",method="GET",status="404",le="+Inf"} 1 public_api_http_requests_duration_seconds_sum{endpoint="/time.php",method="GET",status="404"} 0.000011828 public_api_http_requests_duration_seconds_count{endpoint="/time.php",method="GET",status="404"} 1 # ... public_api_http_requests_total{endpoint="/time.php",method="GET",status="404"} 1
Then, repeat for 100+ other 404'ing endpoints, and you end up with a really large metrics response, resulting in hundreds of GB of egress on your /metrics endpoint (which has been kindly refunded by the Fly.io billing team).
/metrics
Furthermore, a dedicated attacker could certainly exhaust significantly more resources simply by 404ing on more URLs
To resolve this issue, 404 responses from endpoints that fail to match should not be included in metrics. I believe this PR should resolve the issue?
Also recommend submitting an advisory to RustSec for this version of the package.
Just found https://github.com/nlopes/actix-web-prom/pull/71 and exclude_status(actix_http::StatusCode::NOT_FOUND)
exclude_status(actix_http::StatusCode::NOT_FOUND)
I have applied this temporary work around to my repo, but is not in the docs or readme
Currently, when an endpoint receives a 404, it extends the Prometheus metrics response by approx ~15 new lines / ~1.5KB.
For example, my server does not include a
/time.php
endpont, so when I was bot scraped by a bot, of these lines were appended.Then, repeat for 100+ other 404'ing endpoints, and you end up with a really large metrics response, resulting in hundreds of GB of egress on your
/metrics
endpoint (which has been kindly refunded by the Fly.io billing team).Furthermore, a dedicated attacker could certainly exhaust significantly more resources simply by 404ing on more URLs
To resolve this issue, 404 responses from endpoints that fail to match should not be included in metrics. I believe this PR should resolve the issue?
Also recommend submitting an advisory to RustSec for this version of the package.