OpenTelemetry metrics

[thepwagner]

Add support for https://github.com/open-telemetry/opentelemetry-go metrics, with the configuration supporting otlp over http and grpc.

RFC: Configuration schema

The initial version of this PR slapped the configuration into the existing [statsd] section, but since opentelemetry != statsd we opted to make configuration changes:

• The [statsd] section is effectively renamed [metrics]: instead of awkwardly specifying statsd.backend = "otel", you say metrics.backend = "otel".
• Fields specific to the statsd client stay behind in the [statsd] section
• New fields specific to the opentelemtry client are added to a new [opentelemetry] section.

Thanks _rxbchen for the feedback on this layout!

• [x] Once we've settled on the schema, I'll update the README etc with these new flags

Testing instructions

To test this, I started from Shopify's configuration:

$ go run ./cmd/voucher_server -c .../config/staging/config.toml

Within that configuration, I tweaked the [statsd] section:

[statsd]
backend = "otel"
addr = "http://localhost:4318"
# OR: addr = "grpc://localhost:4317"
insecure = true # needed for http and grpc vs Shopify's test collector
tags = ["voucher_env:staging"]

Then I used voucher_client to trigger attestations and verify the metrics.

[thepwagner]

cc @Owen-Cummings

[rxbchen]

imo I like the idea of renaming statsd to metrics. Also having an extra section called [opentelemetry] and [statsd] would make it more clear as to what sorts of metrics we are trying to configure. Just my opinion though!

[thepwagner]

I like the idea of renaming statsd to metrics.

I dig it!

I've updated the documentation to say that [metrics] is now "the way" to specify the backend. The [statsd] section is still supported, so that we're not breaking any existing configs!

• The backend and tags are common to all implementations, so they migrate to the new [metrics] section.
• The addr and sampling_interval fields are specific to statsd, so they stay behind in the [statsd] section.
• The addr, insecure and interval fields are specific to opentelemetry, so they move to the new [opentelemetry] section.

(addr looks like it might be shared here - but it does nothing when `backend = "datadog").

I've updated the PR description, this is ready for another review - PTAL!