search

bitsofinfo / hazelcast-docker-swarm-discovery-spi

Docker Swarm based discovery strategy SPI for Hazelcast enabled applications
Apache License 2.0
39 stars 33 forks source link

SwarmMemberAddressProvider fails with underscore in swarmMgrUri #48

Closed mnoky closed 5 years ago

mnoky commented 5 years ago

Hi, I'm using 1.0-RC14 and am trying to get SwarmMemberAddressProvider working. I'm using the tecnativa/docker-socket-proxy (running as a separate service, configured via swarmMgrUri) to safely proxy the docker socket. Everything seems to be configured correctly, I can see the following in the log message at startup:

org.bitsofinfo.hazelcast.discovery.docker.swarm.SwarmDiscoveryUtil.null SwarmDiscoveryUtil[SwarmMemberAddressProvider].discoverNodes(): via DOCKER_HOST: localhost
docker-network-names = mynetwork
docker-service-names = myservice
docker-service-labels = null
swarmMgrUri = http://docker-socket-proxy_service:2375
skipVerifySsl = false
logAllServiceNamesOnFailedDiscovery = false
strictDockerServiceNameComparison = false

However, then the following exception is generated at startup:


com.hazelcast.config.ConfigurationException: Cannot create a new instance of MemberAddressProvider 'class org.bitsofinfo.hazelcast.discovery.docker.swarm.SwarmMemberAddressProvider'
    at com.hazelcast.instance.DefaultNodeContext.newMemberAddressProviderInstance(DefaultNodeContext.java:101)
    at com.hazelcast.instance.DefaultNodeContext.createAddressPicker(DefaultNodeContext.java:71)
    at com.hazelcast.instance.Node.<init>(Node.java:182)
    at com.hazelcast.instance.HazelcastInstanceImpl.createNode(HazelcastInstanceImpl.java:160)
    at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:128)
    at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:195)
    at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:174)
    at edu.cornell.birds.speciesaccount.config.HazelcastHttpSessionConfig.createHazelcastInstance(HazelcastHttpSessionConfig.java:86)
    at edu.cornell.birds.speciesaccount.config.HazelcastHttpSessionConfig.hazelcastInstance(HazelcastHttpSessionConfig.java:51)
    at edu.cornell.birds.speciesaccount.config.HazelcastHttpSessionConfig$$EnhancerBySpringCGLIB$$4550389d.CGLIB$hazelcastInstance$0(<generated>)
    at edu.cornell.birds.speciesaccount.config.HazelcastHttpSessionConfig$$EnhancerBySpringCGLIB$$4550389d$$FastClassBySpringCGLIB$$f1aea3f0.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
    at edu.cornell.birds.speciesaccount.config.HazelcastHttpSessionConfig$$EnhancerBySpringCGLIB$$4550389d.hazelcastInstance(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1177)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1072)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1131)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.getObject(DefaultListableBeanFactory.java:1649)
    at org.springframework.session.hazelcast.config.annotation.web.http.HazelcastHttpSessionConfiguration.setHazelcastInstance(HazelcastHttpSessionConfiguration.java:96)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:698)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:364)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1268)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:551)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:756)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
    at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
    at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
    at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5236)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:744)
    at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:624)
    at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1833)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException: null
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.hazelcast.instance.DefaultNodeContext.newMemberAddressProviderInstance(DefaultNodeContext.java:94)
    ... 64 common frames omitted
Caused by: java.lang.RuntimeException: SwarmAddressPicker: Error constructing SwarmDiscoveryUtil: SwarmDiscoveryUtil[SwarmMemberAddressProvider].discoverContainers() unexpected error: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: java.lang.IllegalArgumentException: Host name may not be null
    at org.bitsofinfo.hazelcast.discovery.docker.swarm.SwarmMemberAddressProvider.initialize(SwarmMemberAddressProvider.java:298)
    at org.bitsofinfo.hazelcast.discovery.docker.swarm.SwarmMemberAddressProvider.initialize(SwarmMemberAddressProvider.java:202)
    at org.bitsofinfo.hazelcast.discovery.docker.swarm.SwarmMemberAddressProvider.<init>(SwarmMemberAddressProvider.java:179)
    ... 69 common frames omitted
Caused by: java.lang.Exception: SwarmDiscoveryUtil[SwarmMemberAddressProvider].discoverContainers() unexpected error: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: java.lang.IllegalArgumentException: Host name may not be null
    at org.bitsofinfo.hazelcast.discovery.docker.swarm.SwarmDiscoveryUtil.discoverContainers(SwarmDiscoveryUtil.java:392)
    at org.bitsofinfo.hazelcast.discovery.docker.swarm.SwarmDiscoveryUtil.discoverSelf(SwarmDiscoveryUtil.java:173)
    at org.bitsofinfo.hazelcast.discovery.docker.swarm.SwarmDiscoveryUtil.<init>(SwarmDiscoveryUtil.java:167)
    at org.bitsofinfo.hazelcast.discovery.docker.swarm.SwarmMemberAddressProvider.initialize(SwarmMemberAddressProvider.java:294)
    ... 71 common frames omitted
Caused by: com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: java.lang.IllegalArgumentException: Host name may not be null
    at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2848)
    at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2702)
    at com.spotify.docker.client.DefaultDockerClient.listNetworks(DefaultDockerClient.java:2446)
    at org.bitsofinfo.hazelcast.discovery.docker.swarm.SwarmDiscoveryUtil.discoverContainers(SwarmDiscoveryUtil.java:333)
    ... 74 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: java.lang.IllegalArgumentException: Host name may not be null
    at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
    at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2700)
    ... 76 common frames omitted
Caused by: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: java.lang.IllegalArgumentException: Host name may not be null
    at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:178)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
    ... 5 common frames omitted
Caused by: java.lang.IllegalArgumentException: Host name may not be null
    at com.spotify.docker.client.shaded.org.apache.http.util.Args.containsNoBlanks(Args.java:81)
    at com.spotify.docker.client.shaded.org.apache.http.HttpHost.<init>(HttpHost.java:81)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector.getHost(ApacheConnector.java:514)
    at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
    ... 21 common frames omitted

Not really sure what is going on or why a particular hostname is null. Can you offer some insight here?

mnoky commented 5 years ago

Here are some more details I discovered: I am using a swarmMgrUri that looks like this: http://docker-socket-proxy_service:2375. This is the tecnativa/docker-socket-proxy running in a separate stack. Notice the underscore in the name, which is added automatically when the docker stack is deployed (stackName_serviceName). Problem is, the Java URI class doesn't like this! For example, if you run this snippet of code:

URI u1 = new URI("http://docker-socket-proxy_service:2375");
URI u2 = new URI("http://docker-socket-proxy:2375");
System.out.println("u1=" + u1 + " host=" + u1.getHost());
System.out.println("u2=" + u2 + " host=" + u2.getHost());

The result is:

u1=http://docker-socket-proxy_service:2375 host=null
u2=http://docker-socket-proxy:2375 host=docker-socket-proxy

Notice that the host is null when an underscore is used in the name! Thus, when DefaultDockerClient.listNetworks() happens, the hostname is null.

Is there any way to work around this?

mnoky commented 5 years ago

It appears that this is a widespread issue with the java URI class. Although the underscore character is technically disallowed in URIs according to the RFC, it happens all the time due to docker stacks, S3 buckets, etc. The redisson project worked around this by literally hacking the URI class. This is ugly and fragile, but it works:

https://github.com/redisson/redisson/issues/1367 https://github.com/redisson/redisson/commit/dd13ac99e25f148bf24332ee1b9da096e0f9f067

Original source of workaround:

https://stackoverflow.com/questions/28568188/java-net-uri-get-host-with-underscores/28568433

Not sure if you are amenable to adding this workaround in the hazelcast-docker-swarm-discovery-spi project, but it would be helpful. For now, I can add the workaround to my code which instantiates it. In our use cases, we really want to have a single docker-socket-proxy running in a separate stack which can be leveraged by multiple different services in other stacks.

bitsofinfo commented 5 years ago

Glad you got it figured out! If you'd like to contribute a PR to address this, please do so and we can incorporate it in, or at a minimum a good note in the README covering this issue and the workaround.

mnoky commented 5 years ago

It turns out the URI hack is really fragile and doesn't work well in newer versions of java. I found a better solution was to run the docker-socket-proxy using a network alias so that it can be accessed without the use of an underscore. In docker-compose.yml:

services:
  proxy-service:
    networks:
      my-network:
        aliases:
          - dockersocketproxy

This allows me to use a swarmMgrUriof http://dockersocketproxy:2375