Codespaces: "Cannot connect to the Docker daemon" occurs randomly (postCreateCommand)

[mandrasch]

Hi,

I'm trying to get DDEV - an open source Docker-based PHP development toolkit - working reliably in Codespaces. We already had a discussion here, there were some fixes for docker-in-docker in unversal image in the past (🙏 ) and @eiriksm (and others) provided a wait for docker script for the postCreateCommand, which I thought worked.

#!/bin/bash
set -ex

wait_for_docker() {
  while true; do
    docker ps > /dev/null 2>&1 && break
    sleep 1
  done
  echo "Docker is ready."
}

wait_for_docker

# proceed with commands requiring docker

But for some Codespaces launches I still run into

"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?".

Other launches work fine as expected, current ratio is 5 working, 1 error - seems random.

My devcontainer.json:

{
    "image": "mcr.microsoft.com/devcontainers/universal:2",
    "features": {
        "ghcr.io/ddev/ddev/install-ddev:latest": {}
    },
    // ...
    "postCreateCommand": "chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh"
}

Example error output:

2024-05-18 08:23:04.024Z: chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh
2024-05-18 08:23:04.171Z: + wait_for_docker
+ ********
+ docker ps
2024-05-18 08:23:04.666Z: + break
2024-05-18 08:23:04.670Z: + echo 'Docker is ready.'
Docker is ready.
+ ddev debug download-images
2024-05-18 08:23:04.829Z: Downloading ********/docker/compose/releases/download/v2.27.0/docker-compose-linux-x86_64 ... 2024-05-18 08:23:04.839Z: 
2024-05-18 08:23:05.647Z: 
docker-compose 0 B / 60.09 MiB [---------------------------------------]   0.00%2024-05-18 08:23:05.848Z: 
docker-compose 25.86 MiB / 60.09 MiB [==============>------------------]  43.04%2024-05-18 08:23:06.048Z: 
docker-compose 41.45 MiB / 60.09 MiB [======================>----------]  68.99%2024-05-18 08:23:06.235Z: 
docker-compose 60.09 MiB / 60.09 MiB [==============================] 100.00% 0s
Download complete. 
2024-05-18 08:23:20.065Z: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024-05-18 08:23:20.078Z: Failed to PullBaseContainerImages(): exit status 1 
2024-05-18 08:23:20.085Z: {"outcome":"error","message":"Command failed: /bin/sh -c chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh","description":"The postCreateCommand in the devcontainer.json failed.","containerId":"26560cae79c4541c929a677101cf0dea4a9632ceb233c8571c85425a124d2a54"}
2024-05-18 08:23:20.126Z: postCreateCommand failed with exit code 1. Skipping any further user-provided commands.

Full Log: https://gist.github.com/mandrasch/f0ddc7c275c8df68275c88bae5ec9424

My setup: https://github.com/mandrasch/ddev-codespaces-launch-blank Usage: Browser-based postCreateCommand: https://github.com/mandrasch/ddev-codespaces-launch-blank/blob/main/.devcontainer/postCreateCommand.sh

Is there a way to debug this? Thanks very much in advance for assistance! 🤗

[rfay]

I thought this had already been dealt with a time or two in previous issues?

[mandrasch]

I thought this had already been dealt with a time or two in previous issues?

Yes, there were fixes in docker-in-docker (universal image), but there is also a new spec for better lifecycle with docker in codespaces mentioned here: https://github.com/devcontainers/features/issues/634#issuecomment-1692306977

I'm not sure if this improvement would solve the random docker connection error I saw a few times now or if this is unrelated, therefore I opened up a new issue.

[samruddhikhandale]

Hi 👋

Docker is ready. 2024-05-18 08:23:20.065Z: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

This doesn't look good, the wait_for_docker script thinks the docker started, but apparently that's not the case?

Can you update https://github.com/mandrasch/ddev-codespaces-launch-blank/blob/main/.devcontainer/postCreateCommand.sh as follows and see if this helps? Thanks!

# Docker can take a couple seconds to come up. Wait for it to be ready before
# proceeding with bootstrap.
iterations=10
while ! docker ps &>/dev/null; do
  if [[ $iterations -eq 0 ]]; then
    echo "Timeout waiting for the Docker daemon to start."
    exit 1
  fi

  iterations=$((iterations - 1))
  echo 'Docker is not ready. Waiting 10 seconds and trying again.'
  sleep 10
done

[mandrasch]

Hi @samruddhikhandale, thanks very much for reply! Much appreciated!

I tested the new wait for docker script.

1 attempt out of 11 still failed: https://gist.github.com/mandrasch/1d293ba7ebe34aef6b8134a4637b29ad

In the failed attempt, docker ps seems to be responding immediately? There are no iterations, but it fails afterwards.

2024-06-06 15:52:38.406Z: + iterations=102024-06-06 15:52:38.419Z: 
+ docker ps
2024-06-06 15:52:38.657Z: + ddev debug download-images2024-06-06 15:52:38.668Z: 

2024-06-06 15:52:53.999Z: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024-06-06 15:52:54.004Z: Failed to PullBaseContainerImages(): exit status 1 

The sucessful attempts seem to run through some iterations before docker ps responds:

2024-06-06 15:53:07.738Z: chmod +x .devcontainer/postCreateCommand.sh && .devcontainer/postCreateCommand.sh
2024-06-06 15:53:07.850Z: + iterations=10
+ docker ps
2024-06-06 15:53:19.330Z: + [[ 10 -eq 0 ]]2024-06-06 15:53:19.333Z: 
+ iterations=9
+ echo 'Docker is not ready. Waiting 10 seconds and trying again.'
Docker is not ready. Waiting 10 seconds and trying again.
+ sleep 10
2024-06-06 15:53:29.346Z: + docker ps
2024-06-06 15:53:39.589Z: + [[ 9 -eq 0 ]]
2024-06-06 15:53:39.598Z: + iterations=8
+ echo 'Docker is not ready. Waiting 10 seconds and trying again.'
Docker is not ready. Waiting 10 seconds and trying again.
+ sleep 10
2024-06-06 15:53:49.604Z: + docker ps
2024-06-06 15:53:49.641Z: + ddev debug download-images
2024-06-06 15:53:49.726Z: Downloading ********/docker/compose/releases/download/v2.27.0/docker-compose-linux-x86_64 ... 

Log of successful attempt: https://gist.github.com/mandrasch/8b296552af9c66a1acc4036a10f1154b

[samruddhikhandale]

Thanks for getting back to us. @gauravsaini04 Can you help investigate the issue? It would be good to find out what could be causing the docker to stop after getting started.