Retries (or similar) when accessing secrets from secret manager

[aarpabe]

Is your feature request related to a problem? Please describe.

We use secret manager in our spring boot production app. From time to time, we get DEADLINE_EXCEEDED errors when accessing a secret. This is usually a short time problem and if we retried, the issue is gone.

We'd like to be able to avoid these errors with a simple declarative approach (without much boilerplate).

Describe the solution you'd like

• A configuration option to enable automatic retries when fetching credentials.

Describe alternatives you've considered

• Manual implementation of retries
• somehow configuring grpc to retry under the hood (not sure, if possible)

Additional context

• we do not know, if there is a simple way already. If there is one, please let us know
• current version used:

  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>spring-cloud-gcp-starter-secretmanager</artifactId>
    <version>2.0.2</version>
    <scope>compile</scope>
  </dependency>

• stack trace of our error

  
  io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 59.999980428s. [buffered_nanos=1736630, remote_addr=secretmanager.googleapis.com/172.217.23.106:443]

at io.grpc.Status.asRuntimeException ( io/grpc/Status.java:535 ) at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose ( io/grpc.stub/ClientCalls.java:533 ) at io.grpc.internal.ClientCallImpl.closeObserver ( io/grpc.internal/ClientCallImpl.java:553 ) at io.grpc.internal.ClientCallImpl.access$300 ( io/grpc.internal/ClientCallImpl.java:68 ) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal ( io/grpc.internal/ClientCallImpl.java:739 ) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext ( io/grpc.internal/ClientCallImpl.java:718 ) at io.grpc.internal.ContextRunnable.run ( io/grpc.internal/ContextRunnable.java:37 ) at io.grpc.internal.SerializingExecutor.run ( io/grpc.internal/SerializingExecutor.java:123 ) at java.util.concurrent.Executors$RunnableAdapter.call ( java/util.concurrent/Executors.java:515 ) at java.util.concurrent.FutureTask.run ( java/util.concurrent/FutureTask.java:264 ) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run ( java/util.concurrent/ScheduledThreadPoolExecutor.java:304 ) at java.util.concurrent.ThreadPoolExecutor.runWorker ( java/util.concurrent/ThreadPoolExecutor.java:1128 ) at java.util.concurrent.ThreadPoolExecutor$Worker.run ( java/util.concurrent/ThreadPoolExecutor.java:628 ) at java.lang.Thread.run ( java/lang/Thread.java:829 )