search

galasa-dev / projectmanagement

Project Management repo for Issues and ZenHub
7 stars 4 forks source link

IO Exception thrown (intermitently) when loading overrides using etcd3 client (Eclipse Plugin) #367

Closed Jimbo4794 closed 4 years ago

Jimbo4794 commented 4 years ago

Ive encoutnered this a number of times while developing the kubernetes operator, but today the new grad in the CIP team encountered this during a local run. Upon trying to start a test run the exception below is thrown.

I initially thought it was a an error in the ecosystem I was bringing up, espeically with the "could not retreieve key" error, which i was getting from the 'submit automated run' which didnt come with any trace, just the message. Now from the trace provided from CIP I can see its a IOException, but cant determine what exactly is causing it. Somestimes restarting eclipse resolves this, but that wasnt working for Daniella today.

Stack trace:

dev.galasa.boot.LauncherException: dev.galasa.framework.TestRunException: Problem loading overrides from the run properties
    at dev.galasa.boot.felix.FelixFramework.runTest(FelixFramework.java:222)
    at dev.galasa.boot.Launcher.launch(Launcher.java:152)
    at dev.galasa.boot.Launcher.main(Launcher.java:106)
Caused by: dev.galasa.framework.TestRunException: Problem loading overrides from the run properties
    at dev.galasa.framework.TestRunner.runTest(TestRunner.java:132)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at dev.galasa.boot.felix.FelixFramework.runTest(FelixFramework.java:220)
    ... 2 more
Caused by: dev.galasa.framework.spi.DynamicStatusStoreException: Could not retrieve key.
    at dev.galasa.cps.etcd.internal.Etcd3DynamicStatusStore.getPrefix(Etcd3DynamicStatusStore.java:276)
    at dev.galasa.framework.internal.dss.FrameworkDynamicStoreKeyAccess.getPrefix(FrameworkDynamicStoreKeyAccess.java:142)
    at dev.galasa.framework.TestRunner.runTest(TestRunner.java:125)
    ... 7 more
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
    at dev.galasa.cps.etcd.internal.Etcd3DynamicStatusStore.getPrefix(Etcd3DynamicStatusStore.java:263)
    ... 9 more
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
    at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:512)
    at io.etcd.jetcd.ClientConnectionManager.lambda$execute$4(ClientConnectionManager.java:361)
    at net.jodah.failsafe.Functions.lambda$toCtxSupplier$13(Functions.java:272)
    at net.jodah.failsafe.Functions.lambda$getPromise$1(Functions.java:80)
    at net.jodah.failsafe.RetryPolicyExecutor$1.call(RetryPolicyExecutor.java:101)
    at net.jodah.failsafe.RetryPolicyExecutor.lambda$supplyAsync$1(RetryPolicyExecutor.java:139)
    at net.jodah.failsafe.Functions.lambda$null$3(Functions.java:97)
    at net.jodah.failsafe.internal.util.DelegatingScheduler.lambda$schedule$0(DelegatingScheduler.java:140)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
    at io.grpc.Status.asRuntimeException(Status.java:533)
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.etcd.jetcd.ClientConnectionManager$AuthTokenInterceptor$1$1.onClose(ClientConnectionManager.java:311)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    ... 3 more
Caused by: java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:192)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
    at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:247)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1140)
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ... 1 more
21/07/2020 11:09:26.800 DEBUG dev.galasa.boot.felix.FelixFramework.stopFramework - Stopping Felix framework
Exception in thread “main” java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at org.apache.felix.framework.util.ThreadGate.await(ThreadGate.java:79)
    at org.apache.felix.framework.Felix.waitForStop(Felix.java:1195)
    at dev.galasa.boot.felix.FelixFramework.stopFramework(FelixFramework.java:630)
    at dev.galasa.boot.Launcher.launch(Launcher.java:174)
    at dev.galasa.boot.Launcher.main(Launcher.java:106)
21/07/2020 11:09:26.804 INFO FelixStartLevel dev.galasa.framework.Framework.deactivate - Framework service deactivatedweb
Jimbo4794 commented 4 years ago

So this is looking like its a networking (VPN related) issue. Daniella hasnt encoutnered this since changing to MEA, rather than MEA 2 and ive not encountered it since using the America VPN