HEPCloud / decisionengine

HEPCloud Decision Engine framework
Apache License 2.0
6 stars 25 forks source link

Metrics that are declared to have labels need to have them specified at measurement time #698

Open shreyb opened 5 months ago

shreyb commented 5 months ago

In src/decisionengine/framework/engine/DecisionEngine.py, there are a number of metrics that are declared like this that have labels specified, like this one:

https://github.com/HEPCloud/decisionengine/blob/0a38b170be381b3a69e533bd6cb7cc360ae166a9/src/decisionengine/framework/engine/DecisionEngine.py#L460

However, later on in the module, the metric is measured without a label:

https://github.com/HEPCloud/decisionengine/blob/0a38b170be381b3a69e533bd6cb7cc360ae166a9/src/decisionengine/framework/engine/DecisionEngine.py#L1205

@vitodb , @mambelli , and I did some tests and believe this will cause the DE to error out (probably crashing the DE) if the affected command (de-client --set_channel_log_level in this case) is run.

Please correct this and all other metrics as necessary so that if they are declared initially to have a label, that the label is set at measurement time.

Note that in some cases, this might require not using a decorator (since, for example, for the above case, you won't have the source name outside of the function, but you do INSIDE the function), and rather using the context manager as shown here:

https://prometheus.github.io/client_python/instrumenting/gauge/

Let me know if you have any questions about this issue.