hashgraph / hedera-services

Crypto, token, consensus, file, and smart contract services for the Hedera public ledger
Apache License 2.0
310 stars 135 forks source link

Enforce metric name rules #5967

Closed poulok closed 9 months ago

poulok commented 1 year ago

There are currently no restrictions on what characters can be used in metric names. Some characters cause problems for systems like Prometheus and Elastic to ingest. There are currently scripts that modify metric names to make them compatible with other systems.

Metric names should be validated at the time of creation. There are at least two options for this:

  1. Throw an exception when a metric is created with an invalid name
  2. Modify the name of the metric to conform to the standards when publishing to externals

Elastic converter: https://github.com/swirlds/infrastructure/blob/master/hedera-utils/PipelineGenerator.py Prometheus converter: com.swirlds.common.metrics.platform.prometheus.NameConverter

mxtartaglia-sl commented 10 months ago

Noting here to discuss with @tannerjfco:

  1. Elastic Converter: it is not there anymore; where should I look for it? What are the constraints for the converter?
  2. A suggestion is to throw an exception during Metric creation, but should that be accompanied by some change other than the hedera-services codebase?
  3. How can I get Prometheus access?
poulok commented 9 months ago

We decided to log an error if a metric violates a rule, and not throw an exception. The log message will fail nightly tests but not cause problems in a production network.

Additionally, check the unit name for compatibility because the unit is appended to the metric name.

mxtartaglia-sl commented 9 months ago
List of decisions made after discussing with @hendrikebbers   Option 1 Option 2
  Throw exception Log Error
Design  
  Not Required Required for Metrics Supporting Units
Unit  
  suffix of the name Prometheus handle
Unit  
  Fixed values Free values
Unit  
  Create and assert naming conventions Free values
Naming Style  
  Validate In Metrics Framework Validate in Prometheus adapter
Design  
  Extension Performance
Design