Lower default active series in agent integration

[rfratto]

The agent integration exposes ~700 active series out of the box, most of which may not be useful for just scraping agent metrics.

Some extra options should be provided to narrow down the default set of collected metrics to as small as possible.

[rfratto]

Looking at the list of exported metrics, there's quite a few ones that aren't used with the default configs:

• etcd metrics from the KV store API when the KV store API is disabled (e.g., etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket)
• prometheus_* series for service discovery mechanisms not in use

The go instrumentation can also be disabled fairly easily.

The fix for this ideally should:

1. Provide flags for disabling instrumentation (like with the go and Weaveworks server instrumentation)
2. Remove series from being registered if they are not used (like with the etcd metrics)
3. Filter out the remaining set of metrics that are still unwanted

grafana/agent#2 will take a lot of work coordinating upstream PRs, so I'm going to opt for just adding a filtering mechanism for now. The filtering mechanism will wrap a Gatherer and determine whether a series should be removed. The list of unwanted metrics will be defined either:

1. "intelligently" by not collecting metrics for Agent components that are unused, like the etcd metrics
2. By flag, where the user specifies they don't want Prometheus metrics.

As all integrations are embedded, this filtering mechanism can be eventually applied on top of existing exporters.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[rfratto]

There's still a lot of metrics registered by default that we probably don't need. I introduced a TODO comment in grafana/agent#2388 to stop using prometheus.DefaultRegisterer for all of our metrics so we have more control over what gets exposed.

There's probably some metrics registered there that we want to keep, so it's not a simple switch - we'll have to go through the metrics individually and identify what we want to keep and ensure we work with the relevant upstream projects to have those metrics be exposable another way.