Using random ports from portbindings

[jelmew]

Issue Overview

As user of arquillian cube, I would like to be able to use random ports selected by the docker daemon

Current Behaviour

The default port is taken since no remapping is done if cube configuration is like (Cube format):

containerName_*: image: myimage portBindings: [9990/tcp,8080/tcp]

This will try to launch the Docker container with a configuration which is like: containerName_42953_42954_806c5a61-ec20-4d3a-9073-8e7f5a58b097: image: myimage killContainer: false manual: false portBindings: [53607->9990/tcp, 53752->8080/tcp] readonlyRootfs: false removeVolumes: true

After which the port bindings are not passed to the test-code. Which tries to use 9990/8080. This, of course, doesn't succeed.

Expected Behaviour

When running arquillian-cube with random ports, those ports are remapped to the containerConfiguration, so that the tests can run in parallel. So after launching the container(probably event BeforeSuite?) the ports should be rebound if the containerName contains a wildcard.

Additional Information

If I want to implement this feature, what would be the way to go. Is this possible to do so and If it is possible, are there some pointers to which information is available where?

[lordofthejars]

Maybe what you need is https://github.com/arquillian/arquillian-cube/blob/master/docker/ftest-standalone-star-operator-docker-compose/src/test/java/org/arquillian/cube/StandaloneStarOperatorTestCase.java#L28 ? With this the injected port is the random one.

[jelmew]

Point is, the code doesn't even get there. It fails before injecting those fields with org.jboss.arquillian.container.spi.client.container.DeploymentException: Cannot deploy: customer-test.war

[lordofthejars]

But this error happens because of port? Do you have a full stack of the error?

[jelmew]

Yes. If I manually assign a portbinding it connects. So this works: containerName_: //Without wildcard image: myimage portBindings: [53607->9990/tcp, 53752->8080/tcp]

With the random port bindings, I can see it tries to connect to 9990 using wireshark.

The stacktrace is Stacktrace: org.jboss.arquillian.container.spi.client.container.DeploymentException: Cannot deploy: customer-test.war at org.jboss.as.arquillian.container.ArchiveDeployer.deployInternal(ArchiveDeployer.java:141) at org.jboss.as.arquillian.container.ArchiveDeployer.deployInternal(ArchiveDeployer.java:121) at org.jboss.as.arquillian.container.ArchiveDeployer.deploy(ArchiveDeployer.java:83) at org.jboss.as.arquillian.container.CommonDeployableContainer.deploy(CommonDeployableContainer.java:236) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController$3.call(ContainerDeployController.java:161) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController$3.call(ContainerDeployController.java:128) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController.executeOperation(ContainerDeployController.java:271) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController.deploy(ContainerDeployController.java:127) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.container.impl.client.container.DeploymentExceptionHandler.verifyExpectedExceptionDuringDeploy(DeploymentExceptionHandler.java:50) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.container.impl.client.ContainerDeploymentContextHandler.createDeploymentContext(ContainerDeploymentContextHandler.java:78) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.container.impl.client.ContainerDeploymentContextHandler.createContainerContext(ContainerDeploymentContextHandler.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:145) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:116) at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController$1.perform(ContainerDeployController.java:95) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController$1.perform(ContainerDeployController.java:80) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController.forEachDeployment(ContainerDeployController.java:263) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController.forEachManagedDeployment(ContainerDeployController.java:239) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController.deployManaged(ContainerDeployController.java:79) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:145) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:116) at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.execute(ContainerEventController.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:92) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:73) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:96) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:145) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:116) at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.beforeClass(EventTestRunnerAdaptor.java:87) at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:202) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:431) at org.jboss.arquillian.junit.Arquillian.access$200(Arquillian.java:55) at org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:219) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:167) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Iterator.forEachRemaining(Iterator.java:116) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:83) at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:74) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:170) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:154) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:90) at org.junit.platform.console.tasks.ConsoleTestExecutor.executeTests(ConsoleTestExecutor.java:65) at org.junit.platform.console.tasks.ConsoleTestExecutor.lambda$execute$0(ConsoleTestExecutor.java:57) at org.junit.platform.console.tasks.CustomContextClassLoaderExecutor.invoke(CustomContextClassLoaderExecutor.java:33) at org.junit.platform.console.tasks.ConsoleTestExecutor.execute(ConsoleTestExecutor.java:57) at org.junit.platform.console.ConsoleLauncher.executeTests(ConsoleLauncher.java:84) at org.junit.platform.console.ConsoleLauncher.execute(ConsoleLauncher.java:74) at org.junit.platform.console.ConsoleLauncher.execute(ConsoleLauncher.java:47) at org.junit.platform.console.ConsoleLauncher.main(ConsoleLauncher.java:39) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.xnio.nio.WorkerThread$ConnectHandle.handleReady(WorkerThread.java:319) at org.xnio.nio.WorkerThread.run(WorkerThread.java:559) at ...asynchronous invocation...(Unknown Source) at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:294) at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:275) at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:379) at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:367) at org.jboss.as.protocol.ProtocolConnectionUtils.connect(ProtocolConnectionUtils.java:83) at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:114) at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:257) at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:71) at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:212) at org.jboss.as.controller.client.impl.RemotingModelControllerClient.getOrCreateChannel(RemotingModelControllerClient.java:146) at org.jboss.as.controller.client.impl.RemotingModelControllerClient$1.getChannel(RemotingModelControllerClient.java:65) at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:147) at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:122) at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:263) at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:168) at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeAsync(AbstractModelControllerClient.java:111) at org.jboss.as.controller.client.helpers.standalone.impl.ModelControllerClientServerDeploymentManager.executeOperation(ModelControllerClientServerDeploymentManager.java:50) at org.jboss.as.controller.client.helpers.standalone.impl.AbstractServerDeploymentManager.execute(AbstractServerDeploymentManager.java:79) at org.jboss.as.controller.client.helpers.standalone.ServerDeploymentHelper.deploy(ServerDeploymentHelper.java:54) at org.jboss.as.arquillian.container.ArchiveDeployer.deployInternal(ArchiveDeployer.java:135) ... 110 more

[lordofthejars]

Ok, then I think that you can help on this :) nice catch. Currently, I am trying to figure out if there is a big change or not, first class to put a debug point would be: https://github.com/arquillian/arquillian-cube/blob/master/core/src/main/java/org/arquillian/cube/impl/client/container/ProtocolMetadataUpdater.java to check that the port is correct there.

[lordofthejars]

Notice that there is where port mapping is done, (line 44) is where you remap configured port but currently I am not 100% sure if this is where deployment port is resolved.

[lordofthejars]

The other point to look at which I think it is more accurate is this class https://github.com/arquillian/arquillian-cube/blob/8bcf1960df5be7b2536884e4d2d3513ae21e53cb/core/src/main/java/org/arquillian/cube/impl/client/container/ContainerConfigurationController.java where the arquillian configuration object is overrode by the docker ports.

[jelmew]

I'm on vacation soon. I'll look if I can find time to look at it before I go off, otherwise, I'll look at it after I'm back (which is in a month or so).

[lordofthejars]

ok, if not if you can share a project, I can inspect.

[jelmew]

Ok, I'm back from a long vacation. In ContainerConfigurationController, on line 40, this call returns null. Cube<?> cube = cubeRegistry.getCube(ContainerUtil.getCubeIDForContainer(container));

Of course, after that, no remapping is done at all and the function returns. My initial suspect is that this has something to do with the randomized id.

[jelmew]

In LocalCubeRegistry things go wrong. It asks for a cube with Id that is without the added randomized UUID. (So let's say in my case id='customer-domain-test_30274_30275'). The cube is known in the hashmap as customer-domain-test_30274_30275_d4d87b7e-6812-42ab-9ea6-c07acb789c37. No match, returns null.

This is due to the fact that in the ContainerConfigurationController, when it calls: ContainerUtil.getContainerByDeployableContainer(containerRegistry, event.getDeployableContainer()); Line 34, it get's a container with the name customer-domain-test_30274_30275, instead of the randomized one. Any idea where these are set? I believe the UUID is added to the container name after it is registered in the containerRegistry (if that is the correct english term). This I can see when debugging ContainerUtil.getContainerByDeployableContainer (Line 10 of ContainerUtil)

When looking it up in the ContainerRegistryCreator, I can see it happening. In the createRegistry it registers the container with the name without the star operator.

[jelmew]

Ok @lordofthejars , do you have an idea where the star operator is parsed to an uuid? I'm now slowly getting lost in arquillian core and arquillian container code, which is not were our problem is. Did I describe the problem correctly in the comment above?

[jelmew]

So I did some further digging. The createRegistry call in ContainerRegistryCreator is hit and registers the container using the non wildcard even before the UUID(By star operator) is added in the CubeDuckerConfigurator. Which adds it with the UUID to the CubeDockerConfiguration. What would be a proper way to solve this? One would need to make sure that getCube returns the cube that is called, as described in my comment above.

So to summarize, just to make it clear. The code expects that the cubeId matches the containerName. This is not the case since the cubeId differs due to the fact that an UUID is appended to it using the wildcard Operator. Therefore, no attempt at remapping is done. This seems like a bug to me, now just how to solve it. Any suggestions @lordofthejars (p.s sorry for all the spam)

[jelmew]

Hi @lordofthejars . I got a rudimentary fix in that confirmed works (no tests etc). Would this be a valid solution in terms of architecture and code? Or is there a cleaner solution?

[jelmew]

Hello. Any update on this or anything I can help with?

[jelmew]

@lordofthejars (or someone else). This issue has been open for some while now. The current pull request is unsuitable because we don't want to force an container dependency. Now I want to contribute to solve this problem, since this bug is still holding my team back. You said in the pull rquest:

So probably the implementation will require some kind of split so in case of standalone the classes that require the container thing are not loaded.

If some pointers could set me on a path to achieve this split, that would be great. Arquillian-cube is not a small project and juist whacking around in random classes probably won't do much good.

[lordofthejars]

I'll try during this week to take your PR and try to see how this split should be done.

[dipak-pawar]

@jelmew I am looking into this issue and your pr. Can you provide small reproducer with the configuration you are using to reproduce this issue?

[jelmew]

I tested it in private code, I'm trying to make a small example now.

[jelmew]

Ok. Here is a very simple example. The branch master works as expected, an war is deployed to the docker container using the remapped port. The other not_working branch uses the wildcard (check arquillian.xml). The port is not remapped in the arquillian-cube-docker code (see comments above), therefore arquillian tries to deploy the war to 9990, which is not available.

P.S for this to run, expose your 2375 docker port on your localhost.