Deadline exceeded after 3 minutes

[rameshar16]

Hi @stephenh,

I am getting the below error on the client-side. 2020-04-28 19:40:40 INFO Connection status: Status{code=DEADLINE_EXCEEDED, description=deadline exceeded after 179997852647ns, cause=null} 2020-04-28 19:40:40 INFO Stopping session 2020-04-28 19:40:40 ERROR Exception starting the client java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 179997852647ns at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:512) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) at mirror.MirrorClient.startSession(MirrorClient.java:124) at mirror.MirrorClient.access$300(MirrorClient.java:27) at mirror.MirrorClient$SessionStarter.runOneLoop(MirrorClient.java:198) at mirror.tasks.ThreadBasedTask.run(ThreadBasedTask.java:62) at mirror.tasks.ThreadBasedTask.lambda$new$0(ThreadBasedTask.java:39) at java.lang.Thread.run(Thread.java:748) Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 179997852647ns at io.grpc.Status.asRuntimeException(Status.java:533) at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:507) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:627) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:515) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:686) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:675) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 common frames omitted 2020-04-28 19:40:40 INFO Stopping session 2020-04-28 19:40:40 INFO Connected, starting session, version 1.3.6-3-gb47dd9f-dirty

Please help me in fixing this issue.

Thanks and Regards, Ramesh AR

[rameshar16]

Hi @stephenh,

Is this tool works for large amount of files?

Thanks and Regards, Ramesh AR

[stephenh]

I'm currently using mirror with ~500,000 files, which are all source code files, i.e. not large video/binary/etc. files. It works really well for that setup.

How many files are you trying to sync?

It looks like 179997852647 nanos is ~exactly three minutes, so there is probably an RPC timeout that could be adjusted.

Ah, there it is:

https://github.com/stephenh/mirror/blob/9e72dff857d389d6853b535fd85b1122a73c9be6/src/main/java/mirror/Utils.java#L28

You could try bumping that up to, say, 10 minutes and seeing what happens/if that fixes it for you.

[rameshar16]

Hi @stephenh,

Please let me know the way to update the timeout value to 10 min. Is there any way to pass the timeout value from the command line? or Do I need to rebuild the code with 10min?

Thanks and Regards, Ramesh AR

[stephenh]

Yes, right now you'd have to rebuild the code with 10 minutes. I can work on adding a command line flag but would take a few days to get around to.

[rameshar16]

Thank you @stephenh for your quick help.

[rameshar16]

Hi @stephenh, Thank you for the update. I have updated the timeout value to 10 min and ran the build. The build is running for the past 5 hours. Is it expected?

`<=========----> 76% EXECUTING [5h 18m 49s]

:test > Executing test mirror.IntegrationTest`

Thanks and Regards, Ramesh AR

[stephenh]

No, the tests should take ~2-3 minutes. IntegrationTest does take longer b/c it uses the real local file system and real client/server instances, so it needs to do some sleeps etc while doing assertions, so it is not "all in-memory unit test" fast, but definitely not 5 hours.

I've never seen that happen so no good guesses about why, other than try it a few times. :shrug:

[rameshar16]

Hi @stephenh,

Thank you for the update. I have given the "./gradlew clean check shadowJar" to build the jar. It seems the check option is taking more time for me. Later I have give "./gradlew clean shadowJar" and the build completed in mins.

The tool is working fine after I increased the timeout value.

Thank you again for your help.

Thanks and Regards, Ramesh AR