vert-x3 / vertx-config

Vert.x Configuration Service
Apache License 2.0
54 stars 64 forks source link

Kubernetes ConfigMap blocks the event loop #18

Closed InfoSec812 closed 7 years ago

InfoSec812 commented 7 years ago

See example project... When I use the fabric8 maven plugin to deploy this in either our OCP cluster OR using Minishift, I get the following error repeatedly:

 WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 10290 ms, time limit is 2000
io.vertx.core.VertxException: Thread blocked
 at io.vertx.core.impl.BlockedThreadChecker.registerThread(BlockedThreadChecker.java:67)
 at io.vertx.core.impl.VertxThreadFactory.newThread(VertxThreadFactory.java:64)
 at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:612)
 at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:925)
 at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
 at io.vertx.core.impl.TaskQueue.execute(TaskQueue.java:78)
 at io.vertx.core.impl.ContextImpl.executeBlocking(ContextImpl.java:291)
 at io.vertx.core.impl.ContextImpl.executeBlocking(ContextImpl.java:240)
 at io.vertx.core.impl.VertxImpl.executeBlocking(VertxImpl.java:650)
 at io.vertx.core.impl.VertxImpl.executeBlocking(VertxImpl.java:656)
 at io.vertx.config.kubernetes.ConfigMapStore.getClient(ConfigMapStore.java:74)
 at io.vertx.config.kubernetes.ConfigMapStore.get(ConfigMapStore.java:105)
 at io.vertx.config.impl.ConfigurationProvider.get(ConfigurationProvider.java:35)
 at io.vertx.config.impl.ConfigRetrieverImpl.lambda$compute$8(ConfigRetrieverImpl.java:167)
 at io.vertx.config.impl.ConfigRetrieverImpl$$Lambda$32/648252746.apply(Unknown Source)
 at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
 at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
 at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
 at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
 at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
 at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
 at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
 at io.vertx.config.impl.ConfigRetrieverImpl.compute(ConfigRetrieverImpl.java:176)
 at io.vertx.config.impl.ConfigRetrieverImpl.getConfig(ConfigRetrieverImpl.java:98)
 at io.vertx.config.impl.ConfigRetrieverImpl.<init>(ConfigRetrieverImpl.java:82)
 at io.vertx.config.ConfigRetriever.create(ConfigRetriever.java:32)
 at com.redhat.labs.ocp.MainVerticle.start(MainVerticle.java:45)
 at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:434)
 at io.vertx.core.impl.DeploymentManager$$Lambda$26/1932831450.handle(Unknown Source)
 at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
 at io.vertx.core.impl.ContextImpl$$Lambda$27/440737101.run(Unknown Source)
 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445)
 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 at java.lang.Thread.run(Thread.java:748)
cescoffier commented 7 years ago

@InfoSec812 when running your project, I've:


SEVERE: Error loading config
--
  | io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1/api/v1/namespaces/foo/configmaps/ambassador-service. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..

Which is the expected output.

It's a raw minishift in a new project without any particular permission. Do I have to do something to reproduce it ?

InfoSec812 commented 7 years ago

@cescoffier I believe that the issue was demonstrated when there was either:

  1. No corresponding ConfigMap for the requested name
  2. Insufficient permissions

I will try to get you a better answer as soon as I have a few moments.

InfoSec812 commented 7 years ago

I remember now...Failing to specify the namespace is what caused it to block the event loop!

cescoffier commented 7 years ago

Thanks for the info, gonna reproduce it right now...

InfoSec812 commented 7 years ago

Woot! Thanks @cescoffier