To connect a docker host, we need a client-cert.pem and client-key.pem - the paths of these files on the local Brooklyn machine are passed as the identity and credential to jclouds (see DockerHostImpl.preStart(), in the DOCKER_GENERATE_TLS_CERTIFICATES block).
Unfortunately we are passing to jclouds the absolute path of these files, and we are writing these files to /tmp. On rebind (i.e. restarting Brooklyn), the brooklyn location immediately reads these files (to construct the ComputeServiceContext). If the file doesn't exist or is not a valid .pem certificate/key, it throws an exception that causes rebind to report failures.
Using the file system is a bad idea - it breaks HA. It is an even worse idea to use /tmp because all the files can be lost or reboot.
Short term, we should write these files to a more stable directory.
Longer term, we should ensure the contents of these files are stored in persisted state. Ideally we'd pass the cert contents to jclouds, rather than passing the path of a local file.
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /tmp/SDvmZ4wj-cert.pem (No such file or directory)
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.jclouds.docker.suppliers.DockerSSLContextSupplier.get(DockerSSLContextSupplier.java:60)
at org.jclouds.docker.config.DockerOkHttpClientSupplier.get(DockerOkHttpClientSupplier.java:49)
at org.jclouds.docker.config.DockerOkHttpClientSupplier.get(DockerOkHttpClientSupplier.java:30)
at org.jclouds.http.okhttp.config.OkHttpCommandExecutorServiceModule$OkHttpClientProvider.get(OkHttpCommandExecutorServiceModule.java:71)
at org.jclouds.http.okhttp.config.OkHttpCommandExecutorServiceModule$OkHttpClientProvider.get(OkHttpCommandExecutorServiceModule.java:54)
at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:53)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204)
at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:198)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:179)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
at com.google.inject.Guice.createInjector(Guice.java:95)
at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:402)
at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:326)
at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:608)
at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:588)
at org.jclouds.ContextBuilder.build(ContextBuilder.java:581)
at org.apache.brooklyn.location.jclouds.ComputeServiceRegistryImpl.findComputeService(ComputeServiceRegistryImpl.java:143)
at org.apache.brooklyn.location.jclouds.JcloudsLocation.getComputeService(JcloudsLocation.java:524)
at org.apache.brooklyn.location.jclouds.JcloudsLocation.getComputeService(JcloudsLocation.java:519)
at org.apache.brooklyn.location.jclouds.JcloudsLocation.getComputeService(JcloudsLocation.java:513)
at org.apache.brooklyn.location.jclouds.JcloudsSshMachineLocation.init(JcloudsSshMachineLocation.java:147)
at org.apache.brooklyn.core.mgmt.rebind.BasicLocationRebindSupport.addCustoms(BasicLocationRebindSupport.java:110)
at org.apache.brooklyn.core.mgmt.rebind.BasicLocationRebindSupport.addCustoms(BasicLocationRebindSupport.java:40)
at org.apache.brooklyn.core.mgmt.rebind.AbstractBrooklynObjectRebindSupport.reconstruct(AbstractBrooklynObjectRebindSupport.java:65)
at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.reconstructEverything(RebindIteration.java:540)
at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.doRun(RebindIteration.java:242)
at org.apache.brooklyn.core.mgmt.rebind.InitialFullRebindIteration.doRun(InitialFullRebindIteration.java:69)
at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.run(RebindIteration.java:265)
at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebindImpl(RebindManagerImpl.java:552)
at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl$3.call(RebindManagerImpl.java:502)
at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl$3.call(RebindManagerImpl.java:500)
at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:518)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /tmp/SDvmZ4wj-cert.pem (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at com.google.common.io.Files$FileByteSource.openStream(Files.java:126)
at com.google.common.io.Files$FileByteSource.openStream(Files.java:116)
at com.google.common.io.ByteSource$AsCharSource.openStream(ByteSource.java:435)
at com.google.common.io.CharSource.read(CharSource.java:162)
at com.google.common.io.Files.toString(Files.java:367)
at org.jclouds.docker.suppliers.SSLContextBuilder.loadFile(SSLContextBuilder.java:125)
at org.jclouds.docker.suppliers.SSLContextBuilder.clientKeyAndCertificate(SSLContextBuilder.java:60)
at org.jclouds.docker.suppliers.DockerSSLContextSupplier.get(DockerSSLContextSupplier.java:52)
... 80 more
grkvlt
commented
8 years ago
To connect a docker host, we need a
client-cert.pemandclient-key.pem- the paths of these files on the local Brooklyn machine are passed as the identity and credential to jclouds (seeDockerHostImpl.preStart(), in theDOCKER_GENERATE_TLS_CERTIFICATESblock).Unfortunately we are passing to jclouds the absolute path of these files, and we are writing these files to
/tmp. On rebind (i.e. restarting Brooklyn), the brooklyn location immediately reads these files (to construct theComputeServiceContext). If the file doesn't exist or is not a valid .pem certificate/key, it throws an exception that causes rebind to report failures.Using the file system is a bad idea - it breaks HA. It is an even worse idea to use
/tmpbecause all the files can be lost or reboot.Short term, we should write these files to a more stable directory.
Longer term, we should ensure the contents of these files are stored in persisted state. Ideally we'd pass the cert contents to jclouds, rather than passing the path of a local file.