GoogleCloudMetricExporter should use custom.googleapis.com as the metric prefix by default

[yzhaoa]

Based on https://cloud.google.com/monitoring/custom-metrics#identifier, the only allowable metric prefixes are custom.googleapis.com/, external.googleapis.com/user, and external.googleapis.com/prometheus.

Using the current default of workload.googleapis.com, I run into the following error:

Using the following OpenTelemetry manual instrumentation configuration plus manual gRPC server instrumentation using GrpcTelemetry:

    val traceExporter = TraceExporter.createWithDefaultConfiguration()
    val metricsExporter = GoogleCloudMetricExporter.createWithDefaultConfiguration()

    val meterProvider =
      SdkMeterProvider.builder()
        .registerMetricReader(
          PeriodicMetricReader.builder(metricsExporter).setInterval(Duration.ofMinutes(1)).build()
        )
        .setResource(resource)
        .build()
    return OpenTelemetrySdk.builder()
      .setPropagators(
        ContextPropagators.create(
          TextMapPropagator.composite(
            W3CBaggagePropagator.getInstance(),
            W3CTraceContextPropagator.getInstance(),
          )
        )
      )
      .setMeterProvider(meterProvider)
      .setTracerProvider(
        SdkTracerProvider.builder()
          .addSpanProcessor(
            BatchSpanProcessor.builder(traceExporter).setMeterProvider(meterProvider).build()
          )
          .setResource(resource)
          .build()
      )
      .buildAndRegisterGlobal()

I see the following error when running under GCP:

com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Field labels[2].valueType had an invalid value of "INT64": Metric labels for this user-defined metric are required to be of type 'STRING': workload.googleapis.com
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
        at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:574)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:544)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:535)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

Under debugging, I captured the offending CreateMetricDescriptorRequest:

metric_descriptor {
  labels {
    key: "net_peer_name"
  }
  labels {
    key: "net_transport"
  }
  labels {
    key: "rpc_grpc_status_code"
    value_type: INT64
  }
  labels {
    key: "rpc_method"
  }
  labels {
    key: "rpc_service"
  }
  labels {
    key: "rpc_system"
  }
  metric_kind: CUMULATIVE
  value_type: DISTRIBUTION
  unit: "ms"
  description: "The duration of an outbound RPC invocation"
  display_name: "rpc.client.duration"
  type: "workload.googleapis.com/rpc.client.duration"
}
name: "projects/..."

For reference, the meter is defined at: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/3611f11e3966f2b87da262dc8536e92a5c2747a4/instrumentation-api-semconv/src/main/java/io/opentelemetry/instrumentation/api/instrumenter/rpc/RpcClientMetrics.java#L41

[psx95]

Hi @yzhaoa,

TL;DR

• workload.googleapis.com supports only STRING type metrics, but the domain itself exists and is allowed to be used.
• for metrics other than STRING type, we issued a fix #186 which allows you to override the default metric prefix - you should be able to change it to custom.googleapis.com and it should fix your issue.

I see that you have created the Metric Exporter with default config - so you will probably need to use setPrefix to change the default workload.googleapis.com.

For some background on this

You can read the comments in #155 - which cites the exact same issue that you mentioned.

Based on this comment from this issue - workload.googleapis.com is allowed to be used, despite the deprecation.

[psx95]

Hi @yzhaoa, were you able to resolve your issue with the provided information ?

[psx95]

Closing this issue. Feel free to re-open in case the underlying issue is still unresolved.