sinkingpoint / prometheus-gravel-gateway

A Prometheus Aggregation Gateway for FAAS applications
GNU Lesser General Public License v3.0
115 stars 10 forks source link

Pushing a metric with a subset or superset of existing label names causes error #26

Open brettimus opened 1 year ago

brettimus commented 1 year ago

Hey there!

First off, thanks for your work on this. It's nice to see a well-thought-out solution to getting serverless functions' metrics into Prometheus.

I bumped into an issue pushing metrics of this form:

function_calls_total{function="rabbit",objective_name="animalz"} 1
function_calls_total{function="wildRabbit"} 1

As a workaround, I can set the objective_name label to the empty string whenever it does not exist.

It seems like under the hood, your https://github.com/sinkingpoint/openmetrics-parser library wants for all metrics to have the same labels. (Maybe I should open an issue there instead?)

It gets mad when one metric has a superset or subset of the other's label names.

According to the Prometheus docs, setting a label to the emptystring is the same as not setting it at all.

So, to me, the expected behavior here is that

function_calls_total{function="rabbit",objective_name="animalz"} 1
function_calls_total{function="wildRabbit"} 1

would be equivalent to pushing

function_calls_total{function="rabbit",objective_name="animalz"} 1
function_calls_total{function="wildRabbit",objective_name=""} 1

Curious to hear your thoughts!

sinkingpoint commented 1 year ago

Interesting. We sort of assume that labelsets don't change, either up or down. Will take a poke at this this weekend