Closed braunsonm closed 4 years ago
The agent itself states it has Docker Build support, however the agent cannot find the docker daemon socket to run any docker build commands.
I found that this can be fixed by entering the container and running service docker start
It turns out the containers don't automatically start the daemon on their own for some reason.
Note the following:
root@e607d9db05de:/services# cat run-docker.sh
#!/bin/bash
if [ "$DOCKER_IN_DOCKER" = "start" ] ; then
service docker start
echo "Docker daemon started"
firoot@e607d9db05de:/services# echo $DOCKER_IN_DOCKER
start
As you can see, my DOCKER_IN_DOCKER variable is set, however the script did not run on startup.
Hey,
The DOCKER_IN_DOCKER
setting works for me. They could have updated the image or it could be the way I am formatting it.
This is what I have:
teamcity-agent:
image: jetbrains/teamcity-agent:latest
environment:
- SERVER_URL=http://server:8111
- AGENT_NAME=regular_agent
- DOCKER_IN_DOCKER=start
privileged: true
So I no longer see the failed to get default registry endpoint from daemon
error, but I do now see this error:
docker: Error response from daemon: error creating aufs mount to /var/lib/docker/aufs/mnt/3d9374e474dc71e700bc70591563f5d79409088b4a9ea75b7a338fbc1b2c24f8-init: invalid argument. See 'docker run --help'.
Anyone from the Jetbrains team have any idea what might be going on?
@JamesMcMahon I haven't tested it myself but seems it's impossible to use aufs-in-aufs. AFAIR we don't pass outer docker socket into agent, but running another docker service inside agent, so that approach declares some limitations yet provides security.
Thanks for the answer @VladRassokhin, does that mean that the DOCKER_IN_DOCKER
option is broken? If so it may be good to remove the option all together to avoid people putting in cycles to get it to work.
I actually did get a little further down the road then my post above. I was being a dummy and didn't have a volume mount, which is what was causing that error. Once I added the volume mount I got a completely different error, which I sadly don't have one me.
@JamesMcMahon The option doesn't work in aufs possibly. It works with other file systems.
Ah ok, I've never dived into Docker storage drivers. What is the correct option to use to make this work? Or is it basically, works on everything else except aufs?
I use overlay2 and it works fine other than the issue I mentioned above about failing to auto start. You can enter the container and start it yourself and then it's fine.
Same problem here.
The docker service seems to stop after ~20sec, even if I start it manualy.
Using overlay2
(also tested with aufs
)
Docker version 18.03.0-ce, build 0520e24302
# From inside the agent's container
root@f2691c25f7f7:/# service docker status
* Docker is not running
root@f2691c25f7f7:/# service docker start
* Starting Docker: docker [ OK ]
root@f2691c25f7f7:/# service docker status
* Docker is running
root@f2691c25f7f7:/# time docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
real 0m23.089s
user 0m0.010s
sys 0m0.011s
root@f2691c25f7f7:/# service docker status
* Docker is not running
#docker-compose.yml
version: '3.1'
networks:
ext:
volumes:
docker_volumes:
services:
agent:
image: jetbrains/teamcity-agent:2017.2.3
environment:
- DOCKER_IN_DOCKER=start
- SERVER_URL=server.address
volumes:
- docker_volumes:/var/lib/docker
privileged: true
networks:
- ext
Any idea on how to fix this?
Same result with the less secure approach...
version: '3.1'
networks:
ext:
services:
agent:
image: jetbrains/teamcity-agent
environment:
- DOCKER_IN_DOCKER=start
- SERVER_URL=server.address
volumes:
- volumes/temp=/opt/buildagent/temp
- volumes/work=/opt/buildagent/work
networks:
- ext
Logs when running service docker start
time="2018-04-04T20:34:15.007189666Z" level=info msg="libcontainerd: started new docker-containerd process" pid=3002
time="2018-04-04T20:34:15Z" level=info msg="starting containerd" module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0
time="2018-04-04T20:34:15Z" level=info msg="setting subreaper..." module=containerd
time="2018-04-04T20:34:15Z" level=info msg="changing OOM score to -500" module=containerd
containerd: write /proc/3002/oom_score_adj: permission denied
time="2018-04-04T20:34:15.022913374Z" level=error msg="containerd did not exit successfully" error="exit status 1" module=libcontainerd
Failed to connect to containerd: failed to dial "/var/run/docker/containerd/docker-containerd.sock": dial unix:///var/run/docker/containerd/docker-containerd.sock: timeout
/var/run/docker/containerd/docker-containerd.sock
does not exist
root@758c10b37338:/# ls -lah /var/run/docker/containerd/
total 20K
drwx------ 3 root root 4.0K Apr 4 20:36 .
drwx------ 3 root root 4.0K Apr 4 20:18 ..
-rw------- 1 root root 559 Apr 4 20:36 containerd.toml
drwx--x--x 2 root root 4.0K Apr 4 20:18 daemon
-rw-r----- 1 root root 4 Apr 4 20:36 docker-containerd.pid # <--- disapear when docker service stop
Is the path to docker-containerd.sock invalid? can we change it with a config?
I switched back to the secure method
docker-compose down --volumes
cp ../docker-compose.secure.ymp ./docker-compose.yml
docker-compose up
Stangely, the containerd.sock
is present and the docker service is running.
root@9829cc5ff57f:/# ls -lah /var/run/docker/containerd/
total 20K
drwx------ 3 root root 4.0K Apr 4 20:40 .
drwx------ 6 root root 4.0K Apr 4 20:40 ..
-rw------- 1 root root 559 Apr 4 20:40 containerd.toml
drwx--x--x 3 root root 4.0K Apr 4 20:40 daemon
srw-rw---- 1 root root 0 Apr 4 20:40 docker-containerd-debug.sock
-rw-r----- 1 root root 2 Apr 4 20:40 docker-containerd.pid
srw-rw---- 1 root root 0 Apr 4 20:40 docker-containerd.sock
root@9829cc5ff57f:/# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@9829cc5ff57f:/# service docker status
* Docker is running
But when I tried to run a build configuration agains that agent, the build simply froze...
The build log simply print:
[20:42:54]The build is removed from the queue to be prepared for the start
Its been +10min since the build failed, the agent has been removed from Teamcity but the build is still waiting for it to reply.
This is not what I expect from Jetbrain's products... I wonder how I can justify Teamcity's cost to my boss when I have to spend hours to debug it.
Possible solution that worked for me. The documentation on https://hub.docker.com/r/jetbrains/teamcity-agent/ does not tell you why you need all thoses volumes. Turns out they aren't ment to be shared between your agent, each instance should have its own volumes. I simply removed the volumes as we do not care about the warm up time and everything started working.
Based on this issue https://youtrack.jetbrains.com/issue/TW-53769
Final docker-compose
version: '3.1'
networks:
ext:
services:
agent:
image: jetbrains/teamcity-agent:2017.2.3
environment:
- DOCKER_IN_DOCKER=start
- SERVER_URL=server.address
privileged: true
networks:
- ext
@drawm That is clearly shown in the documentation for using this container. It is also common sense for agents to not share their configuration with other agents.
Your replies have no relation to the issue I reported,
@ChaosCA Well, I've tried it myself and cannot reporoduce issue both on local installation (usign dokcer-compose) and GCP (using kubernetes). Here's my docker-compose.yaml content
version : '2'
services:
server:
image: 'jetbrains/teamcity-server:2017.2.3'
ports:
- 8111:8111
environment:
- TEAMCITY_SERVER_MEM_OPTS="-Xmx750m"
volumes:
- ./data:/data/teamcity_server/datadir
- ./logs:/opt/teamcity/logs
agent:
image: 'jetbrains/teamcity-agent:2017.2.3'
environment:
- SERVER_URL=http://server:8111
- AGENT_NAME=full-agent
- DOCKER_IN_DOCKER=start
privileged: true
Could you please share docker logs from agent instance (/var/log/docker.log
)
@ChaosCA The doc only state the volumes should be present, not why they are needed and what to expect if you mishandle them. Running mutiple agents with the provided example result in the same error you have and thus why I am reporting my findings here.
Anyway, good luck with your issue...
Hi! I have the same problem with the latest version of teamcity-docker-agent. Docker fails on stat up even when I trying to start it with 'service docker start' my deployment.yml: `apiVersion: apps/v1 kind: Deployment metadata: name: teamcity-agents-deployment labels: app: teamcity-agent spec: replicas: 3 selector: matchLabels: app: teamcity-agent template: metadata: labels: app: teamcity-agent spec: containers:
/var/log/docker.log contents
time="2019-07-05T13:06:45.570550340Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2019-07-05T13:06:45.570610219Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42034cf90, CONNECTING" module=grpc
time="2019-07-05T13:06:45.570842028Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42034cf90, READY" module=grpc
time="2019-07-05T13:06:45.571007108Z" level=info msg="Loading containers: start."
time="2019-07-05T13:06:45.571236984Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: , error: exec: \"modprobe\": executable file not found in $PATH"
time="2019-07-05T13:06:45.571282871Z" level=warning msg="Running modprobe nf_nat failed with message: , error: exec: \"modprobe\": executable file not found in $PATH" time="2019-07-05T13:06:45.571314375Z" level=warning msg="Running modprobe xt_conntrack failed with message:
, error: exec: \"modprobe\": executable file not found in $PATH"
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.1: can't initialize iptables table nat': Permission denied (you must be root) Perhaps iptables or your kernel needs to be upgraded. (exit status 3)
After some investigation I fixed it with adding: ` securityContext: capabilities: add:
to deployment configuration
Hi, i'm not sure if it's the same problem, but the /services/run-docker.sh script was running fine at container boot, but when i restarted my host server the docker service would start and after a bit it would error out and exit. i solved it by inserting a 'sleep 20' before running the docker service, and that made it work.
#!/bin/bash
if [ "$DOCKER_IN_DOCKER" = "start" ] ; then
rm /var/run/docker.pid 2>/dev/null
sleep 20
service docker start
echo "Docker daemon started"
fi
Hello @hariseldon78 , could you please clarify a bit part when i restarted my host server the docker service would start and after a bit it would error out and exit
. Could you describe your setup a bit more, I'm not sure I understand what can we do about it.
Thanks,
Hello @kir . My environment is this: an aws ec2 server, on it i am running docker. 2 docker containers: teamcity-server-instance and teamcity-agent-instance. The start command is like this:
docker run --name teamcity-server-instance \
-v $HOME/teamcity/data:/data/teamcity_server/datadir \
-v $HOME/teamcity/logs:/opt/teamcity/logs \
-p 8111:8111 \
--restart=always \
jetbrains/teamcity-server
docker run --name teamcity-agent-instance \
-e SERVER_URL="##########:8111" \
-v $HOME/teamcity/teamcity-agent:/data/teamcity_agent/conf \
-v docker_volumes:/var/lib/docker \
--privileged -e DOCKER_IN_DOCKER=start \
--restart=always \
###########/####-teamcity-agent
(the agent image is on my custom repository because i added some packages to it like nodejs and stuff)
Before inserting the sleep 20
that i mentioned i created the two containers with the commands you see, then i tried stopping and restarting with docker stop - docker start
, and everything was working fine, but when i tried restarting the aws ec2 instance i noticed that the 'docker in docker' feature was not working. So i tried docker exec -it teamcity-agent-instance bash
into the agent container, and with /etc/init.d/docker status
it reported the service turned off. With /etc/init.d/docker start
it would start and work fine after that.
So i tried logging into the container as soon as possible after the aws instance rebooted and i saw it was on, and then after some seconds it turned off, with an error message that i don't remember (i could dig into the log if really really needed).
Then i looked here to understand how the dockerindocker service was started and i found the start script, i understood that it was somehow bound to the docker service running in the ec2 server, and so probably it was a boot order problem, added the sleep and boom! now it works every time.
Hello @hariseldon78 ,
Thanks for the details. I'd really appreciate if you provide the error from the log, as it may clarify on why the docker has stopped in the container. We could add the sleep, but it is better to understand why it is needed at all. Also, how did you configure the TeamCity containers to auto-start on the ec2 instance?
Thanks again,
ok, here is the error (i just removed the sleep and it happened the first reboot):
docker exec -it teamcity-agent-instance bash
root@11d706dfe193:/# /etc/init.d/docker status
* Docker is running
root@11d706dfe193:/# /etc/init.d/docker status
* Docker is running
root@11d706dfe193:/# /etc/init.d/docker status
* Docker is running
root@11d706dfe193:/# /etc/init.d/docker status
* Docker is running
root@11d706dfe193:/# /etc/init.d/docker status
* Docker is not running
root@11d706dfe193:/# date
Fri Oct 11 14:41:41 UTC 2019
(i repeated the docker status command about every 2-3 seconds)
this following is the /var/docker.log since the reboot.
time="2019-10-11T14:40:33.161042011Z" level=info msg="libcontainerd: docker-containerd is still running" pid=240
time="2019-10-11T14:40:33.162402970Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2019-10-11T14:40:33.162423878Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2019-10-11T14:40:33.164780453Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0 <nil>}]" module=grpc
time="2019-10-11T14:40:33.164807427Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2019-10-11T14:40:33.164869425Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42035a390, CONNECTING" module=grpc
time="2019-10-11T14:40:53.165142121Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/docker-containerd.sock 0 <nil>}. Err :connection error: desc = \"transport: error while dialing: dial unix:///var/run/docker/containerd/docker-containerd.sock: timeout\". Reconnecting..." module=grpc
time="2019-10-11T14:40:53.165210502Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42035a390, TRANSIENT_FAILURE" module=grpc
time="2019-10-11T14:40:53.165344758Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42035a390, CONNECTING" module=grpc
time="2019-10-11T14:41:13.165472232Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/docker-containerd.sock 0 <nil>}. Err :connection error: desc = \"transport: error while dialing: dial unix:///var/run/docker/containerd/docker-containerd.sock: timeout\". Reconnecting..." module=grpc
time="2019-10-11T14:41:13.165556900Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42035a390, TRANSIENT_FAILURE" module=grpc
time="2019-10-11T14:41:13.165697424Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42035a390, CONNECTING" module=grpc
time="2019-10-11T14:41:33.165721211Z" level=warning msg="Failed to dial unix:///var/run/docker/containerd/docker-containerd.sock: grpc: the connection is closing; please retry." module=grpc
Failed to connect to containerd: failed to dial "/var/run/docker/containerd/docker-containerd.sock": context deadline exceeded
Also, how did you configure the TeamCity containers to auto-start on the ec2 instance?
I used the '--restart always' option at the docker start or docker update.
Hi @hariseldon78 ,
Thanks a lot for the log. I believe the error is related to this issue: https://github.com/docker/for-linux/issues/517. We'll update the docker version used in our image so the probably it will fix the problem with docker daemon stop.
I've filed this bug in our tracker as https://youtrack.jetbrains.com/issue/TW-62466
Best,
This commit should fix the issue.
The following teamcity agent:
Still fails when trying a Docker Build step.
Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). Using system default: https://index.docker.io/v1/ Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?