Today, the code to create metric descriptors is not resolving "recoverable" errors from "non-recoverable" errors. If a service account does not have access to create metric descriptors, then the collector will attempt to create new metric descriptors every time that metric is encountered, not just once-per-instantiation.
Suggest the following:
The err handling logic is updated so that any 4xx HTTP (or unrecoverable gRPC) error leads to updating the cache, preventing further CMD calls.
If necessary, we update CreateMetricDescriptor configuration to allow configuring the setting for how to deal with failures.
Today, the code to create metric descriptors is not resolving "recoverable" errors from "non-recoverable" errors. If a service account does not have access to create metric descriptors, then the collector will attempt to create new metric descriptors every time that metric is encountered, not just once-per-instantiation.
Suggest the following: