search

tensorchord / envd

🏕️ Reproducible development environment
https://envd.tensorchord.ai/
Apache License 2.0
2.04k stars 160 forks source link

bug: timeout 5s: cannot connect to buildkitd #709

Closed ggaaooppeenngg closed 2 years ago

ggaaooppeenngg commented 2 years ago

Description

bootstrap failed.

Reproduction

envd bootstrap --dockerhub-mirror https://docker.mirrors.sjtug.sjtu.edu.cn
WARN[2022-08-02T10:09:49Z] Warning: open /usr/share/bash-completion/completions/envd: permission denied
WARN[2022-08-02T10:09:49Z] Warning: open /usr/local/share/zsh/site-functions/_envd: permission denied
INFO[2022-08-02T10:09:49Z] You may have to restart your shell for autocomplete to get initialized (e.g. run "exec $SHELL")
WARN[2022-08-02T10:09:49Z] start the created contrainer envd_buildkitd   container=envd_buildkitd driver=docker-container image="docker.io/moby/buildkit:v0.10.3" socket=envd_buildkitd
WARN[2022-08-02T10:09:49Z] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
WARN[2022-08-02T10:09:49Z] commandConn.CloseRead: commandconn: failed to wait: signal: terminated
WARN[2022-08-02T10:09:49Z] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
error: timeout 5s: cannot connect to buildkitd

Additional Info

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

gaocegege commented 2 years ago

Could you please show the result of envd --debug bootstrap --dockerhub-mirror https://docker.mirrors.sjtug.sjtu.edu.cn and envd version --detail?

ggaaooppeenngg commented 2 years ago 
envd --debug bootstrap --dockerhub-mirror https://docker.mirrors.sjtug.sjtu.edu.cn
DEBU[2022-08-03T02:44:59Z] /home/ubuntu/.config/envd/id_rsa_envd.pub already present
DEBU[2022-08-03T02:44:59Z] /home/ubuntu/.config/envd/id_rsa_envd already present
DEBU[2022-08-03T02:44:59Z] home manager initialized                      cache-dir=/home/ubuntu/.cache/envd cache-map="map[oh-my-zsh:true]" cache-status=/home/ubuntu/.cache/envd/cache.status config-file=/home/ubuntu/.config/envd/config.envd context="{default [{default docker-container envd_buildkitd}]}" context-file=/home/ubuntu/.config/envd/contexts
DEBU[2022-08-03T02:44:59Z] /home/ubuntu/.config/envd/id_rsa_envd.pub already present
DEBU[2022-08-03T02:44:59Z] /home/ubuntu/.config/envd/id_rsa_envd already present
WARN[2022-08-03T02:44:59Z] Warning: open /usr/share/bash-completion/completions/envd: permission denied
WARN[2022-08-03T02:44:59Z] Warning: open /usr/local/share/zsh/site-functions/_envd: permission denied
INFO[2022-08-03T02:44:59Z] You may have to restart your shell for autocomplete to get initialized (e.g. run "exec $SHELL")
DEBU[2022-08-03T02:44:59Z] bootstrap the buildkitd container
DEBU[2022-08-03T02:44:59Z] commandconn: starting docker with [exec -i envd_buildkitd buildctl dial-stdio]
DEBU[2022-08-03T02:44:59Z] unimplemented call: SetDeadline(2022-08-03 02:45:19.684203976 +0000 UTC m=+20.009502567)
DEBU[2022-08-03T02:44:59Z] unimplemented call: SetDeadline(0001-01-01 00:00:00 +0000 UTC)
WARN[2022-08-03T02:44:59Z] start the created contrainer envd_buildkitd   container=envd_buildkitd driver=docker-container image="docker.io/moby/buildkit:v0.10.3" socket=envd_buildkitd
DEBU[2022-08-03T02:44:59Z] starting buildkitd                            container=envd_buildkitd mirror="https://docker.mirrors.sjtug.sjtu.edu.cn" tag="docker.io/moby/buildkit:v0.10.3"
DEBU[2022-08-03T02:44:59Z] setting buildkit config:
[registry."docker.io"]
    mirrors = ["https://docker.mirrors.sjtug.sjtu.edu.cn"]  container=envd_buildkitd mirror="https://docker.mirrors.sjtug.sjtu.edu.cn" tag="docker.io/moby/buildkit:v0.10.3"
DEBU[2022-08-03T02:44:59Z] container is running, check if it's ready at docker-container://envd_buildkitd...  container=envd_buildkitd driver=docker-container image="docker.io/moby/buildkit:v0.10.3" socket=envd_buildkitd
DEBU[2022-08-03T02:44:59Z] waiting to connect to buildkitd               container=envd_buildkitd driver=docker-container image="docker.io/moby/buildkit:v0.10.3" socket=envd_buildkitd
WARN[2022-08-03T02:44:59Z] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
WARN[2022-08-03T02:44:59Z] commandConn.CloseRead: commandconn: failed to wait: signal: terminated
WARN[2022-08-03T02:44:59Z] commandConn.CloseWrite: commandconn: failed to wait: signal: terminated
DEBU[2022-08-03T02:45:00Z] failed to connect to buildkitd: failed to list workers: Unavailable: connection closed before server preface received
DEBU[2022-08-03T02:45:01Z] commandconn: starting docker with [exec -i envd_buildkitd buildctl dial-stdio]
DEBU[2022-08-03T02:45:01Z] unimplemented call: SetDeadline(2022-08-03 02:45:21.94302001 +0000 UTC m=+22.268318598)
DEBU[2022-08-03T02:45:01Z] unimplemented call: SetDeadline(0001-01-01 00:00:00 +0000 UTC)
DEBU[2022-08-03T02:45:01Z] commandconn (docker):Error response from daemon: Container c8336b49273ee96900862e4c115733d7441db733f21d870111fc846f3e990ebf is not running
DEBU[2022-08-03T02:45:01Z] failed to connect to buildkitd: failed to list workers: Unavailable: connection closed before server preface received
DEBU[2022-08-03T02:45:02Z] failed to connect to buildkitd: failed to list workers: Unavailable: connection closed before server preface received
DEBU[2022-08-03T02:45:03Z] commandconn: starting docker with [exec -i envd_buildkitd buildctl dial-stdio]
DEBU[2022-08-03T02:45:03Z] unimplemented call: SetDeadline(2022-08-03 02:45:23.97765258 +0000 UTC m=+24.302951168)
DEBU[2022-08-03T02:45:03Z] unimplemented call: SetDeadline(0001-01-01 00:00:00 +0000 UTC)
DEBU[2022-08-03T02:45:04Z] commandconn (docker):Error response from daemon: Container c8336b49273ee96900862e4c115733d7441db733f21d870111fc846f3e990ebf is not running
DEBU[2022-08-03T02:45:04Z] failed to connect to buildkitd: failed to list workers: Unavailable: connection closed before server preface received
error: failed to create buildkit client: failed to bootstrap the buildkitd: failed to connect to buildkitd docker-container://envd_buildkitd: timeout 5s: cannot connect to buildkitd
error: timeout 5s: cannot connect to buildkitd
envd version --detail
envd: v0.2.0-alpha.11+62bd2dc
  BuildDate: 2022-07-29T10:01:19Z
  GitCommit: 62bd2dcf37be2c8b811b34ea6d05189250dbdd1f
  GitTreeState: clean
  GoVersion: go1.18.4
  Compiler: gc
  Platform: linux/amd64
  OSType: linux
  OSVersion: 20.04
  KernelVersion: 5.4.0-1029-aws
  DockerHostVersion: 20.10.17
  ContainerRuntimes: [io.containerd.runc.v2,io.containerd.runtime.v1.linux,runc]
  DefaultRuntime: runc
gaocegege commented 2 years ago

The container envd_buildkitd is not running, could you please show docker logs envd_buildkitd?

And you can docker rm envd_buildkitd and rerun envd bootstrap to give it another try after we got the log.

ggaaooppeenngg commented 2 years ago 
❯ docker logs envd_buildkitd
time="2022-06-24T06:20:01Z" level=info msg="auto snapshotter: using overlayfs"
time="2022-06-24T06:20:01Z" level=warning msg="using host network as the default"
time="2022-06-24T06:20:01Z" level=info msg="found worker \"jcgpl43sasc49iok0zvqpvrcc\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:c8336b49273e org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/386]"
time="2022-06-24T06:20:01Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2022-06-24T06:20:01Z" level=info msg="found 1 workers, default=\"jcgpl43sasc49iok0zvqpvrcc\""
time="2022-06-24T06:20:01Z" level=warning msg="currently, only the default worker can be used."
time="2022-06-24T06:20:01Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
time="2022-06-24T06:45:10Z" level=info msg="trying next host" error="failed to do request: Head \"https://https//docker.mirrors.sjtug.sjtu.edu.cn/v2/tensorchord/python/manifests/3.8-ubuntu20.04?ns=docker.io\": dial tcp: lookup https on 172.31.0.2:53: no such host" host="https://docker.mirrors.sjtug.sjtu.edu.cn"
time="2022-06-24T06:52:52Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = process \"pip install --no-warn-script-location numpy\" did not complete successfully: exit code: 2\n"
time="2022-06-24T06:53:51Z" level=info msg="trying next host" error="failed to do request: Head \"https://https//docker.mirrors.sjtug.sjtu.edu.cn/v2/tensorchord/python/manifests/3.8-ubuntu20.04?ns=docker.io\": dial tcp: lookup https on 172.31.0.2:53: no such host" host="https://docker.mirrors.sjtug.sjtu.edu.cn"
time="2022-06-24T06:54:12Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
mkdir: can't create directory '/etc/buildkit': File exists
❯ docker rm envd_buildkitd
envd_buildkitd
❯ envd bootstrap
WARN[2022-08-03T03:11:21Z] Warning: open /usr/share/bash-completion/completions/envd: permission denied
WARN[2022-08-03T03:11:21Z] Warning: open /usr/local/share/zsh/site-functions/_envd: permission denied
INFO[2022-08-03T03:11:21Z] You may have to restart your shell for autocomplete to get initialized (e.g. run "exec $SHELL")
INFO[2022-08-03T03:11:23Z] The buildkit is running at docker-container://envd_buildkitd

it works

gaocegege commented 2 years ago

The log is weird. I will try to reproduce it locally, thanks for the issue!

VoVAllen commented 2 years ago

Is this resolved after upgrade envd?

gaocegege commented 2 years ago

I think so.

ggaaooppeenngg commented 2 years ago

bootstrap succeeded with the latest version.