Cog fails to run on M1 with the `docker-container` buildx driver

On M1 mac, cog uses buildx to build docker images. However, if the buildx builder is of type docker-container, it fails at the "attach labels to the image" step.

To reproduce the problem, from the cog-examples/hello-world directory, run:

docker buildx create \
        --name cog-bug-example \
        --driver docker-container \
        --driver-opt image='moby/buildkit:v0.11.3'
docker buildx use cog-bug-example
cog build

(To reset afterwards, run docker buildx use default)

This results in output:

Building Docker image from environment in cog.yaml as cog-hello-world...
[+] Building 211.9s (20/20) FINISHED
 => [internal] booting buildkit                                                   3.1s
 => => pulling image moby/buildkit:v0.11.3                                        1.2s
 => => creating container buildx_buildkit_cog-bug-example0                        1.9s
 => [internal] load build definition from Dockerfile                              0.1s
 => => transferring dockerfile: 956B                                              0.0s
 => [internal] load .dockerignore                                                 0.1s
 => => transferring context: 2B                                                   0.0s
 => resolve image config for docker.io/docker/dockerfile:1.2                      2.2s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                  0.0s
 => docker-image://docker.io/docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b2fe6  2.0s
 => => resolve docker.io/docker/dockerfile:1.2@sha256:e2a8561e419ab1ba6b2fe6cbdf  0.0s
 => => sha256:2904710679e8cf1b7e70b02e78dc381d89d57be88fa9add570 9.01MB / 9.01MB  1.7s
 => => extracting sha256:2904710679e8cf1b7e70b02e78dc381d89d57be88fa9add57034e56  0.3s
 => [internal] load build definition from Dockerfile                              0.0s
 => [internal] load .dockerignore                                                 0.0s
 => [internal] load metadata for docker.io/library/python:3.8                     1.3s
 => [auth] library/python:pull token for registry-1.docker.io                     0.0s
 => [stage-0 1/6] FROM docker.io/library/python:3.8@sha256:aea73573c5094e3db037  55.4s
 => => resolve docker.io/library/python:3.8@sha256:aea73573c5094e3db037787e1d691  0.0s
 => => sha256:132919f1ef533ac7db410fb13e673ed79c731b3460b07430130896 244B / 244B  0.1s
 => => sha256:b10fd198281a73a88bd542d782b4d2d5d0d84f233d305d3 15.53MB / 15.53MB  10.4s
 => => sha256:18264500740dfbb825d075853637a67404c1da0089bf54f2a5 6.29MB / 6.29MB  3.8s
 => => sha256:d95aaff50cc752dd058870d3b16250c3265f1faad96a8 202.14MB / 202.14MB  45.7s
 => => sha256:c92bc7c6a37bebf8e7309be1bbbb226b82c52ca2679af36 54.59MB / 54.59MB  16.9s
 => => sha256:120eb239658edf0b32a0e7e017916cc0f4cd2b167667609 16.13MB / 16.13MB  13.1s
 => => sha256:b0248cf3e63c73d0e496a67807d056ca41d5e968b61087e 55.05MB / 55.05MB  18.4s
 => => sha256:17ec37baafa58b5b2b3cfc6ac035d2448a50131e04f3bdc257 2.91MB / 2.91MB  1.4s
 => => extracting sha256:b0248cf3e63c73d0e496a67807d056ca41d5e968b61087e8eca2cf4  3.4s
 => => extracting sha256:120eb239658edf0b32a0e7e017916cc0f4cd2b167667609f87d5975  0.8s
 => => extracting sha256:c92bc7c6a37bebf8e7309be1bbbb226b82c52ca2679af3693c1b6ed  3.5s
 => => extracting sha256:d95aaff50cc752dd058870d3b16250c3265f1faad96a84701f33cc7  8.2s
 => => extracting sha256:18264500740dfbb825d075853637a67404c1da0089bf54f2a5a4d37  0.4s
 => => extracting sha256:b10fd198281a73a88bd542d782b4d2d5d0d84f233d305d3b51679b5  0.6s
 => => extracting sha256:132919f1ef533ac7db410fb13e673ed79c731b3460b074301308964  0.0s
 => => extracting sha256:17ec37baafa58b5b2b3cfc6ac035d2448a50131e04f3bdc25765111  0.4s
 => [internal] load build context                                                 0.0s
 => => transferring context: 146.15kB                                             0.0s
 => [stage-0 2/6] RUN --mount=type=cache,target=/var/cache/apt set -eux; apt-ge  32.1s
 => [stage-0 3/6] COPY .cog/tmp/build2186628050/cog-0.0.1.dev-py3-none-any.whl /  0.0s
 => [stage-0 4/6] RUN --mount=type=cache,target=/root/.cache/pip pip install /t  97.5s
 => [stage-0 5/6] WORKDIR /src                                                    0.0s
 => [stage-0 6/6] COPY . /src                                                     0.0s
 => preparing layers for inline cache                                             4.6s
 => exporting to docker image format                                             13.1s
 => => exporting layers                                                           0.0s
 => => exporting manifest sha256:0d61bd6bcb7e2f91a64923ddd0422902b0f9ca474e95138  0.0s
 => => exporting config sha256:8a6010052e0a9f9cb1797bd886bce795d013538f024076ede  0.0s
 => => sending tarball                                                           13.0s
 => importing to docker                                                           1.9s
Adding labels to image...
#1 [internal] load .dockerignore
#1 transferring context: 2B 0.0s done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 94B 0.0s done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/cog-hello-world:latest
#3 ...

#4 [auth] library/cog-hello-world:pull token for registry-1.docker.io
#4 DONE 0.0s

#3 [internal] load metadata for docker.io/library/cog-hello-world:latest
#3 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
------
 > [internal] load metadata for docker.io/library/cog-hello-world:latest:
------
Dockerfile:1
--------------------
   1 | >>> FROM cog-hello-world
--------------------
ERROR: failed to solve: cog-hello-world: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

ⅹ Failed to add labels to image: exit status 1

It seems that the intermediate cog-hello-world image that was built is not found by the docker-container driver in the "Adding labels to image..." step.

replicate / cog

Cog fails to run on M1 with the `docker-container` buildx driver #1026