zsais / go-gin-prometheus

Gin Web Framework Prometheus metrics exporter
MIT License
450 stars 133 forks source link

High cardinality caused by the url label #36

Open prymitive opened 5 years ago

prymitive commented 5 years ago

url label contains the url of every request that was ever made against gin webserver. This can cause very high memory usage for prometheus server. What's worse this is out of control for the operator of a service running go-gin-prometheus and someone sending a high number of requests with random url can easily cause prometheus server scraping that webserver to run out of memory. This is a common misuse of prometheus metrics and there's even a tool written to detect and mitigate the most serious cases of cardinality explosion - https://blog.freshtracks.io/bomb-squad-automatic-detection-and-suppression-of-prometheus-cardinality-explosions-62ca8e02fa32

This is very easy to reproduce with curl:

for i in {1..1000} ; do curl -so /dev/null "localhost:8080/xxx$i" ; done```

Resulting in:

[...]
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx100"} 1
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx10"} 1
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx1"} 1
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx11"} 1
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx12"} 1
gin_requests_total{code="404",handler="main.notFound",host="localhost:8080",method="GET",url="/xxx13"} 1
[...]

This was already reported in #34 and partially fixed in #35 but those changes are not enough. All metrics already have the handler label that points to the request handler method that the request was routed to, this is enough to identify requests for any debugging / performance tracking.

Please remove the url label from metrics.

satwell commented 4 years ago

Another option might be to use c.FullPath() rather than c.Request.URL.Path for the url label. FullPath was recently added in gin v1.5.0 and returns the path from the matched route or empty string for not found. So cardinality should be limited to the number of routes you have.