i/o timeout with local k3d container registry sometimes

[madhavajay]

What happened?
Sometimes deployment fails during the image push phase.

[frontend] error: Head "http://k3d-registry.localhost:12345/v2/openmined/grid-frontend/blobs/sha256:c46a3e39d3b7fb8c1631cf2a1f4bd11b19ce682581a4835ad499c5fa8954a6ed": proxyconnect tcp: dial tcp 192.168.65.1:3128: i/o timeout
[fatal]  error building image k3d-registry.localhost:12345/openmined/grid-frontend:qBPlas: exit status 1

What did you expect to happen instead?
It should push images and retry if there is an issue.

How can we reproduce the bug? (as minimally and precisely as possible)
I am using k3ds and a local registry. I dont know why this happens but it seems like you can just try and it works fine.

Local Environment:

• DevSpace Version: 5.18.5
• Operating System: mac
• ARCH of the OS: AMD64 Kubernetes Cluster:
• Cloud Provider: k3d + docker desktop (local machine)
• Kubernetes Version: k3d version v5.4.3 k3s version v1.23.6-k3s1 (default)

Anything else we need to know?
This could be the fault of k3d or docker or my machine resources but it would be nice if there was a way to "retry" or extend the timeout on the devspace side.

/kind bug

[FabianKramm]

@madhavajay thanks for creating this issue! I'm not sure if we can implement a retry mechanism there as it seems to be a very unspecific error and retrying does not always solve the problem. With the new pipelines feature you could actually implement this yourself:

functions:
  build_images:
    while ! __build_images "$@"
    do
      echo "Building failed, will try again in 5 seconds..."
      sleep 5
    done

[madhavajay]

@FabianKramm oh thats quite interesting, im trying to understand that syntax, so you can hook the existing steps and call them with some kind of __ syntax or something?

I am totally fine with a solution like that and love the flexibility, when is a good time to upgrade, it seems like maybe its still too soon?

I solved this on my end using glorious one liner ala stackoverflow: https://stackoverflow.com/questions/33353847/retry-a-sequence-of-commands-in-bash-for-a-limited-number-of-times

(r=3;while ! some_cmd ; do ((--r))||exit;sleep 60;done)