Add Configurable Random Sampling to Telemetry

[mkcp]

Feature Description

It is simple to add random sampling of metric emissions. In reloadable config, we can take an int between 0 and 100 (default 100) representing a percent and store it on the telemetry client. On each metric request, we can generate a number between 0 and 100, if the number is less than or equal than the sampling percent, we emit the metric.

For a simplified example,

sampler := 20 // 20% or 1 in 5 requests.

func IncCounter(key, val) {
  if sampler == 100 {
    doIncCounter(key, val)
    return 
  }
  s := rand.Intn(100)
    if s <= sampler {
      doIncCounter(key, val) 
      return
   }
   return
}

Use Case(s)

Sampling is a simple feature that gives both users and maintainers control over metrics activity within an agent. This can be useful for performance, efficiency, and debugging. In an ideal world we'd never have to sample measurements, but it's a valuable option to offer.

[mkcp]

It may be possible to implement this quickly within consul with #8418. Otherwise we can add it as a feature to https://github.com/armon/go-metrics and pass configuration through from:

telemetry {
  sampling = "50%" // or just an int
}