callosin
commented
5 years ago Hi,
I am experimenting the same problem with the v0.3 tag revision. For some reason, it started failling like 2 weeks ago without changing anything.
jul. 01, 2019 9:51:37 A. M. com.google.common.util.concurrent.AggregateFuture$RunningState handleException GRAVE: Got more than one input Future failure. Logging failures after the first com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9999419276ns at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:51) at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:507) at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:482) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9999419276ns at io.grpc.Status.asRuntimeException(Status.java:526) ... 22 more
any idea?
mkuthan
vthornheart-bng
We are using PubSub connector for Kafka Connect from the following commit: https://github.com/GoogleCloudPlatform/pubsub/commit/7d9d4b6f47bf3b87d12b51b6e8319f74a3e829be
The other day we had a problem with one of our network links. This particular link is used to route data between one of our data centres and GCP.
The problem with the link started at 10:18PM and and after tens of seconds the routing has been changed with BGP to use another link and the traffic between our data centre and GCP has been restored. Nonetheless PubSub connector in our Kafka cluster kept reporting DEADLINE_EXCEEDED errors for another half an hour until 10:48PM when it suddenly started working properly again.
During that time we observed increased number of message publish operations in Stackdriver monitoring for published topics (presumably retries). I have checked with our network team and the original link was not restored until later that night.
What is mysterious to us is why this error kept happening after the link switchover and why it resolved itself half an hour later although not much else has changed - at least to our knowledge. We have a hypothesis that after the link switchover the connection to PubSub still kept the old TCP session open and publisher did not receive publish confirmation. Then maybe at 10:48PM the session was forcefully closed at one of the network gateways in between (30 mins seems like a pretty often used session close time) and published has to reconnect, opened a new session and it was publishing uninterrupted since then.
I know it is a difficult issue to reproduce but maybe it also happened to you and there are some hidden flags which we could use but even do not know about to make reconnections in such cases smooth.