metrics: gen-metrics command does not collect all possible metrics

[angles-n-daemons]

Describe the problem After some discussion, it seems that the cockroach gen metric-list command is meant to capture and document all possible metrics which can be omitted by the system, but as it stands currently this isn't the case. Metrics like security.certificate.* and seconds_until_enterprise_license_expiry do not appear in the [metrics page](https://www.cockroachlabs.com/docs/stable/metrics

To Reproduce To validate this, you can run cockroach gen metric-list | grep security to verify that no security metrics are captured. lem.

Environment:

• Current master 8e43568
• OSX

Additional context Unsure the exact % right now, but as it stands the discoverability and documentation of some metrics in the system is hampered by this.

Jira issue: CRDB-42864

[blathers-crl[bot]]

Hi @angles-n-daemons, please add branch-* labels to identify which branch(es) this C-bug affects.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[angles-n-daemons]

Notes: After some investigation, I've found that metrics must be registered to be surfaced and some types of metrics require direct measurement. For example, a labelled counter will not appear in _status/vars unless it's incremented or set beforehand.

With the above information believe that there are four possible reasons why metrics would not appear in the documentation:

• The configuration used to run the server for documentation (TestServerArgs) do not init parts of the system which would register certain metrics.
• It's possible there is some race condition, metrics are scraped for documentation when the service is ready for requests, but it could be possible that at that point metrics have not been registered or measured.
• It's possible (but I feel less so) that the scraping script is incorrectly discarding certain metrics.