Unable to start the elastic agent

[DavidGOrtega]

Macos Catalina 10.15.1 Docker Desktop Version 2.0.0.3 (31259)

I have set everything as discussed. I have even set the enviroment variables AGENT_AUTO_REGISTER_KEY and GO_SERVER_URL not shown in the docs. I have set the job with the client id, but nothing happens

[DavidGOrtega]

Im running the server in a docker container, must be the network

[arvindsv]

Anything in the plugin logs?

[DavidGOrtega]

@arvindsv thanks,

this only

2019-12-04 00:50:13,737 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] Request for Config Migration Started...
2019-12-04 00:50:13,745 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] No Plugin Settings are configured. Skipping Config Migration...
2019-12-04 00:52:52,793 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] Request for Config Migration Started...
2019-12-04 00:52:52,798 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] No Plugin Settings are configured. Skipping Config Migration...
2019-12-04 01:22:08,326 WARN  [qtp1931192767-41] DockerPlugin:97 - Missing docker certificates, will attempt to connect without certificates
2019-12-04 02:20:16,235 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] Request for Config Migration Started...
2019-12-04 02:20:16,240 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] No Plugin Settings are configured. Skipping Config Migration...
2019-12-04 02:37:05,992 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] Request for Config Migration Started...
2019-12-04 02:37:06,008 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] No Plugin Settings are configured. Skipping Config Migration...
2019-12-04 10:49:49,655 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] Request for Config Migration Started...
2019-12-04 10:49:49,670 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] No Plugin Settings are configured. Skipping Config Migration...
2019-12-04 11:32:00,883 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] Request for Config Migration Started...
2019-12-04 11:32:00,887 INFO  [Thread-80] DockerPlugin:72 - [Migrate Config] No Plugin Settings are configured. Skipping Config Migration...

To install it I just dropped the jar in plugins/external and restarted, following then the tutorial.

[arvindsv]

You shouldn't need those environment variables. They're automatically set up by the plugin. I'd expect the plugin to log some messages in the job's console log. Maybe @adityasood has some ideas?

[DavidGOrtega]

You shouldn't need those environment variables.

yeah, I can see it in the code, I started to write plugin using this and swarm as reference

[adityasood]

Apologies I seemed to have missed this thread.

@DavidGOrtega do you see any errors in the GoCD server logs when you try to do this? You mentioned you are running the GoCD server inside a docker container, for the plugin to be able to spin up the docker agents, it will need docker sock mount as well. I tried this on my mac box

docker run --rm -d -it -p8253:8153 -p8254:8154 -v /var/run/docker.sock:/var/run/docker.sock \
  -e GOCD_PLUGIN_INSTALL_docker-elastic-agents=https://github.com/gocd-contrib/docker-elastic-agents-plugin/releases/download/v3.0.0-238-exp/docker-elastic-agents-3.0.0-238.jar gocd/gocd-server:v19.10.0

And could get things working. Would you mind giving it a try and let us know?

[DavidGOrtega]

Sure, let me give it a try

[SamiMe]

I have same problem, any fix?

[GaneshSPatil]

Hello Everyone,

Here are the steps I followed and could verify that the plugin is working as expected. Can you try the following instructions to setup GoCD on docker?

@DavidGOrtega @SamiMe @arvindsv

1. Run GoCD server v20.2.0 with Docker Elastic Agent Plugin installed.

docker run --name gocd-server -d --rm -it -p  8153:8153 -v /var/run/docker.sock:/var/run/docker.sock -e GOCD_PLUGIN_INSTALL_docker-elastic-agents=https://github.com/gocd-contrib/docker-elastic-agents-plugin/releases/download/v3.0.0-245/docker-elastic-agents-3.0.0-245.jar gocd/gocd-server:v20.2.0

The above command will do the following things:

• Pull the official GoCD Docker Image gocd/gocd-server:v20.2.0 if its not present.
• Start a new container.
• Publish container's 8153 port on 8153 port of the host.
• Bind docker socket inside container.
• Install the Docker Elastic Agent Plugin 3.0.0-245.

2. Verify Docker Elastic Agent Plugin is installed.

Once the server is started, verify the plugin is installed by visiting http://localhost:8153/go/admin/plugins.

3. Define Elastic Agent Configurations

Define a new Elastic Agent Configuration to dynamically start agents when relevant GoCD jobs are triggered. Know more about GoCD Elastic Agents here.

Before configuring, find the Server IP address by running the following command:

docker inspect --format='{{(index (index .NetworkSettings.IPAddress))}}' gocd-server

This IP Address will be required in configuring the Go Server URL field.

• Visit http://localhost:8153/go/admin/elastic_agent_configurations
• Click Add. A wizard will be shown to configure an elastic agent.
• On Cluster Profile Tab, configure the following properties:
  • Cluster Name = local-docker-cluster
  • Go Server URL = http://<SERVER_IP_ADDRESS>:8153/go. Replace SERVER_IP_ADDRESS with the GoCD Server container IP address.
  • Agent auto-register Timeout = 5
  • Maximum docker containers to run at any given point in time = 10
  • Docker URI = unix:///var/run/docker.sock
• Hit Save and Next, you will be taken to Elastic Profile tab.
• On Elastic Profile Tab, configure the following properties:
  • Elastic Profile Name = alpine-agent
  • Docker image = gocd/gocd-agent-alpine-3.11:v20.2.0
• Click Finish.

4. Verify Cluster Configurations

Click the Status Report button of the local-docker-cluster to see the cluster status report loads up fine. The status report page should show basic docker cluster information such as CPU, Memory, etc.

5. Create a Pipeline to use the newly configured elastic agent.

• Visit Pipeline Creation Page http://localhost:8153/go/admin/pipelines/create?group=defaultGroup
• Specify the following configurations of the pipeline:
  • Repository URL = https://github.com/GaneshSPatil/junk
  • Pipeline Name = pipeline1
  • Stage Name = stage1
  • Job Name = job-requiring-docker-agent
  • Task = sleep 30
• Click Save + Edit Full Config. Please note that the simplified pipeline creation wizard does not provide an ability to configure a job to use an elastic agent, hence by clicking Save + Edit Full Config, you will be redirected to full configuration page of the pipeline.
• Navigate to job-requiring-docker-agent job's Job Settings Tab.
• Configure Elastic Profile Id as alpine-agent. Make sure that the alpine-agent is shown as an auto-suggestion for the Elastic Profile Id input.
• Click Save.

6. Trigger Pipeline

• Visit Dashboard, verify pipeline1 is shown on the dashboard.
• Unpause pipeline1. Wait for it to automatically start building.
• Once triggered, visit job details page of the pipeline http://localhost:8153/go/tab/build/detail/pipeline1/1/stage1/1/job-requiring-docker-agent
  • Verify the console tab shows the agent starting related logs:

    [go] Received request to create a container of gocd/gocd-agent-alpine-3.11:v20.2.0 at 2020-03-30 06:16:59 +00:00
    [go] Creating container: c3909f5b-d151-41ac-b9ff-3e215f163386
    [go] Starting container: c3909f5b-d151-41ac-b9ff-3e215f163386
    [go] Started container: c3909f5b-d151-41ac-b9ff-3e215f163386

• Once the Started Container log line is printed, Click the Agent Status Report button to verify the agent details are shown.

• Verify the agent is shown on the Cluster status report page by visiting http://localhost:8153/go/admin/status_reports/cd.go.contrib.elastic-agent.docker/cluster/local-docker-cluster

7. Verify Agent Cleanup

Verify that the agent is terminated after the job is completed.

8. Optionally verify the plugin logs:

docker exec gocd-server cat /godata/logs/plugin-cd.go.contrib.elastic-agent.docker.log

[SamiMe]

Good instructions, but i got stuck in step 4. Status Report "gives Failed to load status report" I even clean my Docker environment, but no luck.

Elastic-agent logs(only two):

2020-03-30 11:01:33,402 INFO  [Thread-78] DockerPlugin:72 - [Migrate Config] Request for Config Migration Started...
2020-03-30 11:01:33,413 INFO  [Thread-78] DockerPlugin:72 - [Migrate Config] No Plugin Settings are configured. Skipping Config Migration...

go-server logs: Lots of "Permission denied" errors(part of errors)

Caused by: java.lang.RuntimeException: com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.io.IOException: Permission denied
    at cd.go.contrib.elasticagents.docker.DockerPlugin.handle(DockerPlugin.java:107)
    at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134)
    at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:208)
    ... 11 common frames omitted
Caused by: com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.io.IOException: Permission denied
    at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2848)
    at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2712)
    at com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:594)
    at cd.go.contrib.elasticagents.docker.DockerClientFactory.createClient(DockerClientFactory.java:73)
    at cd.go.contrib.elasticagents.docker.DockerClientFactory.docker(DockerClientFactory.java:46)
    at cd.go.contrib.elasticagents.docker.DockerContainers.docker(DockerContainers.java:174)
    at cd.go.contrib.elasticagents.docker.DockerContainers.refreshAll(DockerContainers.java:140)
    at cd.go.contrib.elasticagents.docker.DockerPlugin.refreshInstancesForCluster(DockerPlugin.java:123)
    at cd.go.contrib.elasticagents.docker.DockerPlugin.refreshInstancesForAllClusters(DockerPlugin.java:113)
    at cd.go.contrib.elasticagents.docker.DockerPlugin.handle(DockerPlugin.java:68)
    ... 13 common frames omitted
Caused by: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.io.IOException: Permission denied
    at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
    at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2710)
    ... 21 common frames omitted
Caused by: javax.ws.rs.ProcessingException: java.io.IOException: Permission denied
    at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
    at org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
    at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
    at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487)
    at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:178)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
    at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ... 1 common frames omitted
Caused by: java.io.IOException: Permission denied
    at jnr.unixsocket.UnixSocketChannel.doConnect(UnixSocketChannel.java:127)
    at jnr.unixsocket.UnixSocketChannel.connect(UnixSocketChannel.java:136)
    at jnr.unixsocket.UnixSocketChannel.connect(UnixSocketChannel.java:223)
    at com.spotify.docker.client.UnixConnectionSocketFactory.connectSocket(UnixConnectionSocketFactory.java:85)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
    at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
    ... 21 common frames omitted

[GaneshSPatil]

Hi @SamiMe - It looks like the container user does not have the execute permissions on /var/run/docker.sock.

Can you check what permissions does /var/run/docker.sock by executing

$ ls -al /var/run/docker.sock

For me, it shows this:

$ ls -al /var/run/docker.sock
lrwxr-xr-x  1 root  daemon  68 Mar 27 12:45 /var/run/docker.sock -> /Users/ganeshp/Library/Containers/com.docker.docker/Data/docker.sock

The last rwxr-xr-x means everyone can execute. The permission model for you might be different.

[SamiMe]

Thank you, it solved the problem. Now Docker agent works.

[GaneshSPatil]

Glad to know that it worked. I will close this issue now. If there are any issues with this plugin, feel free open an issue.