search

GoogleCloudPlatform / opentelemetry-operations-java

Apache License 2.0
71 stars 41 forks source link

Getting "DEADLINE_EXCEEDED: deadline exceeded after 11.999518524s" #343

Closed damjad closed 3 months ago

damjad commented 3 months ago

I am using this lib in Google Cloud Function for creating custom metrics.

I am using Scala. Here's my code https://github.com/damjad/opentelemetry-intro-gcf-scala/blob/06a6abd3ce139cafb92d7eff62a2beeb83be1b98/src/main/scala/com/danish/world/Telemetry.scala

I am facing the DEADLINE_EXCEEDED error intermittently after 11s even though I have specified the deadline to 60s in my code https://github.com/damjad/opentelemetry-intro-gcf-scala/blob/06a6abd3ce139cafb92d7eff62a2beeb83be1b98/src/main/scala/com/danish/world/Telemetry.scala#L34

Here is the full stack trace.

Exporter threw an Exception
com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 11.999518524s. [closed=[], open=[[buffered_nanos=12199509692, waiting_for_connection]]]
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:94)
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
    at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1127)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
    at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:568)
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:570)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
        at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1729)
        at com.google.cloud.monitoring.v3.MetricServiceClient.createTimeSeries(MetricServiceClient.java:1661)
        at com.google.cloud.opentelemetry.metric.CloudMetricClientImpl.createTimeSeries(CloudMetricClientImpl.java:40)
        at com.google.cloud.opentelemetry.metric.InternalMetricExporter.lambda$export$0(InternalMetricExporter.java:219)
        at com.google.cloud.opentelemetry.metric.InternalMetricExporter.createTimeSeriesBatch(InternalMetricExporter.java:250)
        at com.google.cloud.opentelemetry.metric.InternalMetricExporter.export(InternalMetricExporter.java:222)
        at com.google.cloud.opentelemetry.metric.GoogleCloudMetricExporter.export(GoogleCloudMetricExporter.java:90)
        at io.opentelemetry.sdk.metrics.export.PeriodicMetricReader$Scheduled.doRun(PeriodicMetricReader.java:167)
        at io.opentelemetry.sdk.metrics.export.PeriodicMetricReader$Scheduled.run(PeriodicMetricReader.java:153)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        ... 3 more
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 11.999518524s. [closed=[], open=[[buffered_nanos=12199509692, waiting_for_connection]]]
    at io.grpc.Status.asRuntimeException(Status.java:537)
    ... 14 more
punya commented 3 months ago

Hi @damjad. We recommend that customers who want to create custom metrics from Cloud Functions use logs-based metrics.