Add Prometheus support for Cache Providers

go-spatial / tegola

Tegola is a Mapbox Vector Tile server written in Go

http://tegola.io/

MIT License

1.28k stars 194 forks source link

Add Prometheus support for Cache Providers #756

Closed gdey closed 3 years ago

gdey commented 3 years ago

The Cache metrics will be prefixed with tegola_cache_ and the following things will be observed

Each one will have a label of type that will be the cache type name

misses_total is a counter of the total number of misses
hits_total is a counter of the total number of hits
flight_request is a gauge of the cache requests currently being served
duration_seconds is a histogram of the duration of the current get cache requests, partitioned by map_name/map_layer, and z.
response_size_bytes is a histogram of response sized in bytes with the following buckets, 500kb, 1 KB, 10 KB, and 1 MB.
errors is the number of errors from a get operation (that is not a miss error)
set_duration_seconds is a histogram of the duration for filling the cache after a miss.
set_errors is the number of errors from a get operation

gdey commented 3 years ago

@johngian which cache providers you think I should make sure to implement first. And if the above makes sense.

ARolek commented 3 years ago

@gdey I wonder if we should adjust the response_size_bytes a bit to include a 500kb bucket as 500kb is a warning threshold for tile size. We could potentially drop the 5+mb and bucket anything over 1MB

johngian commented 3 years ago

@gdey I think a good start is to add support for the fs based backend and s3.

johngian commented 3 years ago

@gdey Can you ellaborate a bit on the duration_seconds metric? Is it the time for the cache get operation?

Some suggestions for the cache metrics:

Cache get time histogram
Cache set time histogram
Errors counter
Different labels for cache operations on
- tegola serve
- tegola cache seed
- tegola cache purge

gdey commented 3 years ago

@johngian for the tegola cache sub command I would assume we would need to setup Prometheus push type configuration?

ARolek commented 3 years ago

@gdey that's a good point as the cache seed / purge commands don't have a server component, there's no /metric endpoint available. For seeding / purging maybe it makes more sense to analyze the log output?

johngian commented 3 years ago

Can't we have a configuration for that and maybe allow exposing the /metrics endpoint for the cache subcommand as well.

gdey commented 3 years ago

@johngian we could. The thing is the cache sub-command at current does not spin up a web server, it runs and then exits. No webserver involved.

ARolek commented 3 years ago

@gdey we could add a push_url to the observer config to allow for ephemeral job reporting.

gdey commented 3 years ago

Different labels for cache operations on
* `tegola serve`

* `tegola cache seed`

* `tegola cache purge`

wouldn't you set these labels in the Prometheus configs?

johngian commented 3 years ago

As long as its configurable then that's fine. My point is mostly about being able to differentiate the metrics between different use cases.

ARolek commented 3 years ago

@gdey

wouldn't you set these labels in the Prometheus configs?

Do you know how this would work for push? I have not checked into how Prometheus configs work for ephemeral jobs.