The charmstore is tagging charmstore_handler_request_duration with an endpoint tag containing the path of the request. This is bad because it creates a separate series for each distinct path. For example: https://pastebin.canonical.com/p/7Wzv3DVT5v/
Since each charm and bundle has several paths and http methods associated with it, we end up with that many series on this metric. webops reports this metric currently has 33k series.
Prometheus is not really equipped to handle that many series. If we need this for analytics on charm usage, it would be better to label these kind of measurements on influxdb metrics (influx labels are not indexed) or perform a continuous rollup calculation on a stream of this kind of data elsewhere. In the meantime, I recommend removing the endpoint tag from this metric -- or using the abstract route with path variables as a tag value, which will not scale by the number of unique API request inputs.
The charmstore is tagging
charmstore_handler_request_durationwith anendpointtag containing the path of the request. This is bad because it creates a separate series for each distinct path. For example: https://pastebin.canonical.com/p/7Wzv3DVT5v/Since each charm and bundle has several paths and http methods associated with it, we end up with that many series on this metric. webops reports this metric currently has 33k series.
Prometheus is not really equipped to handle that many series. If we need this for analytics on charm usage, it would be better to label these kind of measurements on influxdb metrics (influx labels are not indexed) or perform a continuous rollup calculation on a stream of this kind of data elsewhere. In the meantime, I recommend removing the endpoint tag from this metric -- or using the abstract route with path variables as a tag value, which will not scale by the number of unique API request inputs.