concourse container fails to start

[stathismor]

This is what I see when running `docker-compose-up':

concourse_1     | {"timestamp":"2021-09-21T12:21:20.608056758Z","level":"info","source":"atc","message":"atc.cmd.start","data":{"session":"1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:20.922738927Z","level":"info","source":"atc","message":"atc.cmd.finish","data":{"duration":71094,"session":"1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:20.929029683Z","level":"info","source":"baggageclaim","message":"baggageclaim.using-driver","data":{"driver":"overlay"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:20.929369431Z","level":"info","source":"tsa","message":"tsa.listening","data":{}}
concourse_1     | {"timestamp":"2021-09-21T12:21:20.929509410Z","level":"info","source":"baggageclaim","message":"baggageclaim.listening","data":{"addr":"127.0.0.1:7788"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:20.930535115Z","level":"info","source":"atc","message":"atc.listening","data":{"debug":"127.0.0.1:8079","http":"0.0.0.0:8080"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:20.935110055Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.forwarded-tcpip","data":{"bind-addr":"0.0.0.0:7777","bound-port":32957,"command":"forward-worker","remote":"127.0.0.1:33114","session":"1.4.1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:20.935135730Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.forwarded-tcpip","data":{"bind-addr":"0.0.0.0:7788","bound-port":33359,"command":"forward-worker","remote":"127.0.0.1:33114","session":"1.4.1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:20.935152727Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.start","data":{"command":"forward-worker","remote":"127.0.0.1:33114","session":"1.4.1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:20.935163804Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.register.start","data":{"command":"forward-worker","remote":"127.0.0.1:33114","session":"1.4.1.2","worker-address":"127.0.0.1:32957","worker-platform":"linux","worker-tags":""}}
concourse_1     | {"timestamp":"2021-09-21T12:21:20.935491687Z","level":"error","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.failed-to-dial","data":{"addr":"127.0.0.1:7777","error":"dial tcp 127.0.0.1:7777: connect: connection refused","network":"tcp","session":"4.1.4"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.709445301Z","level":"info","source":"guardian","message":"guardian.no-port-pool-state-to-recover-starting-clean","data":{}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.709934973Z","level":"info","source":"guardian","message":"guardian.metrics-notifier.starting","data":{"interval":"1m0s","session":"5"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.709951210Z","level":"info","source":"guardian","message":"guardian.start.starting","data":{"session":"6"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.710012768Z","level":"info","source":"guardian","message":"guardian.metrics-notifier.started","data":{"interval":"1m0s","session":"5","time":"2021-09-21T12:21:21.71001118Z"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.711025524Z","level":"info","source":"guardian","message":"guardian.cgroups-tmpfs-already-mounted","data":{"path":"/sys/fs/cgroup"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.711072182Z","level":"info","source":"guardian","message":"guardian.mount-cgroup.started","data":{"path":"/sys/fs/cgroup/cpuset","session":"7","subsystem":"cpuset"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.711211092Z","level":"info","source":"guardian","message":"guardian.start.completed","data":{"session":"6"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.711227773Z","level":"error","source":"guardian","message":"guardian.starting-guardian-backend","data":{"error":"bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted"}}
concourse_1     | bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted
concourse_1     | bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714083706Z","level":"error","source":"worker","message":"worker.garden.gdn-runner.logging-runner-exited","data":{"error":"exit status 1","session":"1.2"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714131694Z","level":"error","source":"worker","message":"worker.garden-runner.logging-runner-exited","data":{"error":"Exit trace for group:\ngdn exited with error: exit status 1\n","session":"8"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714198245Z","level":"info","source":"worker","message":"worker.container-sweeper.sweep-cancelled-by-signal","data":{"session":"6","signal":2}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714241576Z","level":"info","source":"worker","message":"worker.container-sweeper.logging-runner-exited","data":{"session":"13"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714212975Z","level":"info","source":"worker","message":"worker.debug-runner.logging-runner-exited","data":{"session":"10"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714216480Z","level":"info","source":"worker","message":"worker.healthcheck-runner.logging-runner-exited","data":{"session":"11"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714213926Z","level":"info","source":"worker","message":"worker.volume-sweeper.sweep-cancelled-by-signal","data":{"session":"7","signal":2}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714292383Z","level":"info","source":"worker","message":"worker.volume-sweeper.logging-runner-exited","data":{"session":"14"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714220880Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.run.context-done","data":{"command":"forward-worker","context-error":{},"session":"4.1.3"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714219796Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.cancelled","data":{"addr":"127.0.0.1:7777","network":"tcp","session":"4.1.4"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714390822Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.run.signal-sent","data":{"command":"forward-worker","session":"4.1.3"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714223084Z","level":"info","source":"worker","message":"worker.baggageclaim-runner.logging-runner-exited","data":{"session":"9"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714531570Z","level":"error","source":"tsa","message":"tsa.connection.channel.command.register.failed-to-fetch-containers","data":{"command":"forward-worker","error":"Get \"http://127.0.0.1:32957/containers\": EOF","remote":"127.0.0.1:33114","session":"1.4.1.2"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714854200Z","level":"error","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.failed-to-dial","data":{"addr":"127.0.0.1:7788","error":"dial tcp 127.0.0.1:7788: connect: connection refused","network":"tcp","session":"4.1.5"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714886770Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.forward-conn.cancelled","data":{"addr":"127.0.0.1:7788","network":"tcp","session":"4.1.5"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.714983967Z","level":"error","source":"tsa","message":"tsa.connection.channel.command.register.failed-to-list-volumes","data":{"command":"forward-worker","error":"Get \"http://127.0.0.1:33359/volumes\": EOF","remote":"127.0.0.1:33114","session":"1.4.1.2"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715004444Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.register.failed-to-reach-worker","data":{"baggageclaim-took":"453.153µs","command":"forward-worker","garden-took":"779.375127ms","remote":"127.0.0.1:33114","session":"1.4.1.2"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715017505Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.register.done","data":{"command":"forward-worker","remote":"127.0.0.1:33114","session":"1.4.1.2","worker-address":"127.0.0.1:32957","worker-platform":"linux","worker-tags":""}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715028955Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.done","data":{"command":"forward-worker","remote":"127.0.0.1:33114","session":"1.4.1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715038832Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.draining-forwarded-connections","data":{"command":"forward-worker","remote":"127.0.0.1:33114","session":"1.4.1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715065899Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.forward-process-exited","data":{"bind-addr":"0.0.0.0:7788","bound-port":33359,"command":"forward-worker","remote":"127.0.0.1:33114","session":"1.4.1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715083273Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.forward-process-exited","data":{"bind-addr":"0.0.0.0:7777","bound-port":32957,"command":"forward-worker","remote":"127.0.0.1:33114","session":"1.4.1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715187110Z","level":"info","source":"worker","message":"worker.beacon-runner.beacon.exited","data":{"session":"4.1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715211737Z","level":"info","source":"worker","message":"worker.beacon-runner.logging-runner-exited","data":{"session":"12"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715229413Z","level":"error","source":"quickstart","message":"quickstart.worker-runner.logging-runner-exited","data":{"error":"Exit trace for group:\ngarden exited with error: Exit trace for group:\ngdn exited with error: exit status 1\n\ncontainer-sweeper exited with nil\ndebug exited with nil\nvolume-sweeper exited with nil\nhealthcheck exited with nil\nbaggageclaim exited with nil\nbeacon exited with nil\n","session":"2"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715290306Z","level":"info","source":"web","message":"web.tsa-runner.logging-runner-exited","data":{"session":"2"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715316216Z","level":"info","source":"atc","message":"atc.tracker.drain.start","data":{"session":"24.1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715352635Z","level":"info","source":"atc","message":"atc.tracker.drain.waiting","data":{"session":"24.1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.715358037Z","level":"info","source":"atc","message":"atc.tracker.drain.done","data":{"session":"24.1"}}
concourse_1     | error: Exit trace for group:
concourse_1     | worker exited with error: Exit trace for group:
concourse_1     | garden exited with error: Exit trace for group:
concourse_1     | gdn exited with error: exit status 1
concourse_1     | 
concourse_1     | container-sweeper exited with nil
concourse_1     | debug exited with nil
concourse_1     | volume-sweeper exited with nil
concourse_1     | healthcheck exited with nil
concourse_1     | baggageclaim exited with nil
concourse_1     | beacon exited with nil
concourse_1     | 
concourse_1     | web exited with nil
concourse_1     | 
concourse_1     | {"timestamp":"2021-09-21T12:21:21.716688577Z","level":"info","source":"web","message":"web.atc-runner.logging-runner-exited","data":{"session":"1"}}
concourse_1     | {"timestamp":"2021-09-21T12:21:21.716738587Z","level":"info","source":"quickstart","message":"quickstart.web-runner.logging-runner-exited","data":{"session":"1"}}

[JasonTheMain]

Hello @stathismor , I ran into a similar issue and was able to fix it by running a different docker-compose.yml Could you please switch it out with

curl -O https://concourse-ci.org/docker-compose.yml

and tell me the result

[emil-nasso]

@JasonTheMain I ran into the same issue and can confirm that it works great with the docker-compose file that you linked instead of the one provided by the tutorial.

[stathismor]

Didn't report here, sorry. I had "solved" the issue by rolling back my cgroup. In Arch, I did this by changing GRUB_CMDLINE_LINUX_DEFAULT in grub config to contain systemd.unified_cgroup_hierarchy=0.

An updated working docker-compose file is definitely preferable. I am not working on concourse at the moment and need to restart my computer to test, so @emil-nasso 's comment should be enough :smile: Thanks both!

[JasonTheMain]

Closing Issue.

[Niceplace]

@stathismor Hey I am looking around because I am also having trouble starting concourse with Docker and I saw another thread in concourse`s "official" docker repository that shows very similar log errors.

Maybe this can help instead of having to downgrade your cgroups ? https://github.com/concourse/concourse-docker/issues/74#issuecomment-925058923

Edit: confirmed that this also fixed my problem with a worker that was failing to start, it seems that might have been the same problem that you had as well.

[charles-dyfis-net]

Experiencing this same issue with Docker on current NixOS 21.11, I worked around it by adding the environment variable:

- CONCOURSE_WORKER_RUNTIME=containerd

...to docker-compose.yml. It would certainly be a Good Thing to have this not be an issue out-of-the-box.

[mattwelke]

I ran into this issue when starting the tutorial too. Adding the env var - CONCOURSE_WORKER_RUNTIME=containerd to the docker-compose.yml file fixed it for me.