search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
7.87k stars 1.09k forks source link

Images created with buildx sometimes have layers that are incorrect/zero bytes #2198

Closed mbentley closed 2 years ago

mbentley commented 3 years ago

Cross posting from https://github.com/docker/buildx/issues/637 as it's probably more relevant directly to buildkit.

I've been building images with buildx primary for the past couple of weeks and I have seen some very odd behavior where the image manifest appears to be written incorrectly. So far, this only appears to happen when I am re-running a build where at least a portion of the image is cached. Here is an example scenario but it doesn't just happen for this one project. Another user reported the same problem on another image with a build that is done using the exact same template as reported here. Here is the GitHub repo for the app I am showing the logs for below.

docker version:

# docker version
Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:40 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Here are the buildx settings:

Creating the builder:

docker buildx create --name builder1 --config ~/buildkit.toml --driver-opt network=host --node builder1

For build buildkit.toml, it just includes values to use my registry cache/mirror to avoid rate limits of Docker Hub:

[registry."docker.io"]
  mirrors = ["registry-mirror.casa.mbentley.net"]

Here is my docker buildx ls & docker buildx inspect output:

$ docker buildx ls
NAME/NODE  DRIVER/ENDPOINT                            STATUS  PLATFORMS
builder1   docker-container
  builder1 unix:///var/lib/jenkins/docker/docker.sock running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
default *  docker
  default  default                                    running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

$ docker buildx inspect builder1
Name:   builder1
Driver: docker-container

Nodes:
Name:      builder1
Endpoint:  unix:///var/lib/jenkins/docker/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Image used by the buildx container:

moby/buildkit:buildx-stable-1@sha256:171689e43026533b48701ab6566b72659dd1839488d715c73ef3fe387fab9a80

I have also tried using from master just in case there was something that has been fixed since then and I have seen the same issue:

moby/buildkit:master@sha256:e0b50ede98f8d241d051b09fceae8956d0e07656657fcd86421e0feff04838ad

I am seeing some messages in the buildx builder container and I am not sure what the errors exactly mean but they don't seem to specifically indicate that something is wrong as they happen when an image was built fine:

...
time="2021-06-20T07:00:11Z" level=warning msg="invalid image config with unaccounted layers"
time="2021-06-20T07:00:12Z" level=warning msg="failed to update distribution source for layer sha256:5c126ace4b8e4d5e2d1fa6699ac82dddc270a48ac66d40ecca9bbccf7e61d697: content digest sha256:5c126ace4b8e4d5e2d1fa6699ac82dddc270a48ac66d40ecca9bbccf7e61d697: not found"
time="2021-06-20T07:00:12Z" level=warning msg="failed to update distribution source for layer sha256:53380879c22c14c31df822b3976e5f7dc41d082e46dc33b5821183449cdd6be3: content digest sha256:53380879c22c14c31df822b3976e5f7dc41d082e46dc33b5821183449cdd6be3: not found"
time="2021-06-20T07:00:13Z" level=warning msg="reference for unknown type: application/vnd.buildkit.cacheconfig.v0"
time="2021-06-20T07:00:21Z" level=warning msg="failed to update distribution source for layer sha256:aa4b46473fcf70f92312bdb4920f0c4c342327d0db0329763a1d1751106fe362: content digest sha256:aa4b46473fcf70f92312bdb4920f0c4c342327d0db0329763a1d1751106fe362: not found"
time="2021-06-20T07:00:21Z" level=warning msg="failed to update distribution source for layer sha256:d960726af2bec62a87ceb07182f7b94c47be03909077e23d8226658f80b47f87: content digest sha256:d960726af2bec62a87ceb07182f7b94c47be03909077e23d8226658f80b47f87: not found"
time="2021-06-20T07:00:21Z" level=warning msg="failed to update distribution source for layer sha256:31e53a1e85966a7f2a77ed3627df81b3bf248754c8434e504426fcf4cf8c982f: content digest sha256:31e53a1e85966a7f2a77ed3627df81b3bf248754c8434e504426fcf4cf8c982f: not found"
time="2021-06-20T07:00:21Z" level=warning msg="failed to update distribution source for layer sha256:3bcb7e2703c0b6d494dcddb5ac5904498385a9f4495a886bc2262651fd2644d4: content digest sha256:3bcb7e2703c0b6d494dcddb5ac5904498385a9f4495a886bc2262651fd2644d4: not found"
time="2021-06-20T07:00:21Z" level=warning msg="failed to update distribution source for layer sha256:2e97a69f7f22afda6401e1fcc261f1e8cf243aeafb5a5217f895fc8f40660467: content digest sha256:2e97a69f7f22afda6401e1fcc261f1e8cf243aeafb5a5217f895fc8f40660467: not found"
time="2021-06-20T07:00:21Z" level=warning msg="reference for unknown type: application/vnd.buildkit.cacheconfig.v0"
...

Here is an example of a build command where I am using the same basic structure across many projects:

docker buildx build \
  --builder builder1 \
  --pull \
  --push \
  --progress plain \
  --build-arg AIRSONIC_VER=$(wget -q -O - https://api.github.com/repos/airsonic/airsonic/releases/latest | jq -r .tag_name) \
  --platform linux/amd64 \
  -t mbentley/airsonic:latest \
  -f Dockerfile \
  --cache-from=type=registry,ref=registry.casa.mbentley.net/mbentley/airsonic:latest-cache \
  --cache-to=type=registry,ref=registry.casa.mbentley.net/mbentley/airsonic:latest-cache,mode=max \
  .

My registry is just a simple v2 open source registry with valid SSL certs for https.

An example error that you might see is something like this:

# docker start airsonic
Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/entrypoint.sh": stat /entrypoint.sh: no such file or directory: unknown
Error: failed to start containers: airsonic

Rebuilding the image appears to re-write the manifest and it works fine. Here are some outputs from a few commands showing the issue on these two images:

Broken: mbentley/airsonic@sha256:e78d073b03e7802217825e7763d33ebf5f07ed4d2adc0bd94d4345854a15d4c3

Working: mbentley/airsonic@sha256:29aa0bb26757325405da60b4aa7d92fa3cf0fb489982ed4c19bc3580ad94ada1

docker history of broken dh_output.txt:

IMAGE          CREATED      CREATED BY                                      SIZE      COMMENT
fa972d2adb4a   4 days ago   CMD ["java" "-Dserver.address=0.0.0.0" "-Dse…   0B        buildkit.dockerfile.v0
<missing>      4 days ago   ENTRYPOINT ["/entrypoint.sh"]                   0B        buildkit.dockerfile.v0
<missing>      4 days ago   VOLUME [/data]                                  0B        buildkit.dockerfile.v0
<missing>      4 days ago   EXPOSE map[4040/tcp:{}]                         0B        buildkit.dockerfile.v0
<missing>      4 days ago   WORKDIR /var/airsonic                           0B        buildkit.dockerfile.v0
<missing>      4 days ago   USER airsonic                                   0B        buildkit.dockerfile.v0
<missing>      4 days ago   COPY entrypoint.sh /entrypoint.sh # buildkit    0B        buildkit.dockerfile.v0
<missing>      4 days ago   RUN |1 AIRSONIC_VER=v10.6.2 /bin/sh -c (mkdi…   0B        buildkit.dockerfile.v0
<missing>      4 days ago   RUN |1 AIRSONIC_VER=v10.6.2 /bin/sh -c (mkdi…   0B        buildkit.dockerfile.v0
<missing>      4 days ago   RUN |1 AIRSONIC_VER=v10.6.2 /bin/sh -c (AIRS…   0B        buildkit.dockerfile.v0
<missing>      4 days ago   RUN |1 AIRSONIC_VER=v10.6.2 /bin/sh -c (mkdi…   4.71kB    buildkit.dockerfile.v0
<missing>      4 days ago   ENV AIRSONIC_MAJOR_VER=10                       0B        buildkit.dockerfile.v0
<missing>      4 days ago   ARG AIRSONIC_VER                                0B        buildkit.dockerfile.v0
<missing>      4 days ago   RUN /bin/sh -c (apk --no-cache add ca-certif…   181MB     buildkit.dockerfile.v0
<missing>      4 days ago   MAINTAINER Matt Bentley <mbentley@mbentley.n…   0B        buildkit.dockerfile.v0
<missing>      4 days ago   /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      4 days ago   /bin/sh -c #(nop) ADD file:f278386b0cef68136…   5.6MB

docker history of working dh_output2.txt:

IMAGE          CREATED      CREATED BY                                      SIZE      COMMENT
2f71e4b4043a   4 days ago   CMD ["java" "-Dserver.address=0.0.0.0" "-Dse…   0B        buildkit.dockerfile.v0
<missing>      4 days ago   ENTRYPOINT ["/entrypoint.sh"]                   0B        buildkit.dockerfile.v0
<missing>      4 days ago   VOLUME [/data]                                  0B        buildkit.dockerfile.v0
<missing>      4 days ago   EXPOSE map[4040/tcp:{}]                         0B        buildkit.dockerfile.v0
<missing>      4 days ago   WORKDIR /var/airsonic                           0B        buildkit.dockerfile.v0
<missing>      4 days ago   USER airsonic                                   0B        buildkit.dockerfile.v0
<missing>      4 days ago   COPY entrypoint.sh /entrypoint.sh # buildkit    865B      buildkit.dockerfile.v0
<missing>      4 days ago   RUN |1 AIRSONIC_VER=v10.6.2 /bin/sh -c (mkdi…   168B      buildkit.dockerfile.v0
<missing>      4 days ago   RUN |1 AIRSONIC_VER=v10.6.2 /bin/sh -c (mkdi…   15B       buildkit.dockerfile.v0
<missing>      4 days ago   RUN |1 AIRSONIC_VER=v10.6.2 /bin/sh -c (AIRS…   84.7MB    buildkit.dockerfile.v0
<missing>      4 days ago   RUN |1 AIRSONIC_VER=v10.6.2 /bin/sh -c (mkdi…   4.71kB    buildkit.dockerfile.v0
<missing>      4 days ago   ENV AIRSONIC_MAJOR_VER=10                       0B        buildkit.dockerfile.v0
<missing>      4 days ago   ARG AIRSONIC_VER                                0B        buildkit.dockerfile.v0
<missing>      4 days ago   RUN /bin/sh -c (apk --no-cache add ca-certif…   181MB     buildkit.dockerfile.v0
<missing>      4 days ago   MAINTAINER Matt Bentley <mbentley@mbentley.n…   0B        buildkit.dockerfile.v0
<missing>      4 days ago   /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      4 days ago   /bin/sh -c #(nop) ADD file:f278386b0cef68136…   5.6MB

In case it is helpful, here are the docker inspect outputs: Broken Working

I am using Jenkins to perform my builds. I see nothing to indicate why this is failing:

Broken build:

+ docker buildx build --builder builder1 --pull --push --progress plain --build-arg AIRSONIC_VER=v10.6.2 --platform linux/amd64 -t mbentley/airsonic:latest -f Dockerfile --cache-from=type=registry,ref=registry.casa.mbentley.net/mbentley/airsonic:latest-cache --cache-to=type=registry,ref=registry.casa.mbentley.net/mbentley/airsonic:latest-cache,mode=max .
#1 [internal] load build definition from Dockerfile
#1 sha256:8df12d67db120c22f2cd768148b423f1e8db4bac2fa3d77864d01a0b90e8d2eb
#1 transferring dockerfile: 2.27kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:ba34cb6f62f35c9741fb8b46667ff98c3ec347a1c1461c4294cb6081f05d292c
#2 transferring context: 2B done
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6
#3 DONE 5.5s

#4 importing cache manifest from registry.casa.mbentley.net/mbentley/airsonic:latest-cache
#4 sha256:54d6ce93e51693820c800f2f73075a6183237f347857e154552181a870bbb9d1
#4 DONE 0.0s

#5 [1/8] FROM docker.io/library/alpine:latest@sha256:234cb88d3020898631af0ccbbcca9a66ae7306ecd30c9720690858c1b007d2a0
#5 sha256:83cc88488a3bbdd9d2e20ed4b0eb8ecd26a4dff43878c2e68a842fed290ad078
#5 resolve docker.io/library/alpine:latest@sha256:234cb88d3020898631af0ccbbcca9a66ae7306ecd30c9720690858c1b007d2a0 0.0s done
#5 DONE 0.0s

#11 [internal] load build context
#11 sha256:d4647f1f5d80ffc907a5589365df0ab882d64d7f2a5ad2e945093014bfc1f410
#11 transferring context: 907B 0.0s done
#11 DONE 0.0s

#12 [7/8] COPY entrypoint.sh /entrypoint.sh
#12 sha256:1c45220cbed07bd67cd26c7e52bce81257cd0eee7ee6be4a94cb3e515a92b92d
#12 CACHED

#7 [3/8] RUN (mkdir /var/airsonic &&  addgroup -g 504 airsonic &&  adduser -h /var/airsonic -D -u 504 -g airsonic -G airsonic -s /sbin/nologin airsonic &&  chown -R airsonic:airsonic /var/airsonic)
#7 sha256:1dd82e7179d413ef03ab329f8e1daf6d9d82624ebb6124f071fd8a3690381720
#7 CACHED

#10 [6/8] RUN (mkdir /data &&  cd /data &&  mkdir db index16 lucene2 lastfmcache thumbs music Podcast playlists .cache .java &&  touch airsonic.properties rollback.sql &&  cd /var/airsonic &&  ln -s /data/db &&  ln -s /data/index16 &&  ln -s /data/lucene2 &&  ln -s /data/lastfmcache &&  ln -s /data/thumbs &&  ln -s /data/music &&  ln -s /data/Podcast &&  ln -s /data/playlists &&  ln -s /data/.cache &&  ln -s /data/.java &&  ln -s /data/airsonic.properties &&  ln -s /data/rollback.sql &&  chown -R airsonic:airsonic /data)
#10 sha256:537ff7293a632ef34d77a985eca7229badd7d2b9309d5e5e11e78ce6cb790cbf
#10 CACHED

#6 [2/8] RUN (apk --no-cache add ca-certificates ffmpeg ttf-dejavu openjdk8 wget jq)
#6 sha256:8eaba8e869c391b7166657cc0a1455b604737370729a18d94ef373a087941341
#6 CACHED

#8 [4/8] RUN (AIRSONIC_VER="$(wget -q -O - https://api.github.com/repos/airsonic/airsonic/releases/latest | jq -r .tag_name)" &&  if [ "$(echo v10.6.2 | awk -F '.' '{print $1}')" != "v10" ]; then echo "Latest version number is no longer 10"; exit 1; fi &&  wget "https://github.com/airsonic/airsonic/releases/download/v10.6.2/airsonic.war" -O /var/airsonic/airsonic.war &&  chown airsonic:airsonic /var/airsonic/airsonic.war)
#8 sha256:3835982beb295b97e3aa4974aca74b0ba79f245ab2e64d5c62615c58cbf37051
#8 CACHED

#9 [5/8] RUN (mkdir /var/airsonic/transcode &&  ln -s /usr/bin/ffmpeg /var/airsonic/transcode/ffmpeg &&  chown -R airsonic:airsonic /var/airsonic/transcode)
#9 sha256:ad84205011c43d7cd451e5f369174eb417683b646df092e20fd67b7a4f745a5a
#9 CACHED

#13 [8/8] WORKDIR /var/airsonic
#13 sha256:cb1ba2e204c69fdfd1ec2b5618052a48c8c9457e19355452ae95b325ec244d09
#13 CACHED

#14 exporting to image
#14 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#14 exporting layers done
#14 exporting manifest sha256:e78d073b03e7802217825e7763d33ebf5f07ed4d2adc0bd94d4345854a15d4c3 0.0s done
#14 exporting config sha256:fa972d2adb4a1fcc619a9ae6c02662c74ba0b564ef40675a0a52d40b110cd21c 0.0s done
#14 pushing layers
#14 ...

#15 [auth] mbentley/airsonic:pull,push token for registry-1.docker.io
#15 sha256:af6bfb8d65f9654c0328f15c4fcd08bd98be5645b915ca1105506ee7e5ea8467
#15 DONE 0.0s

#14 exporting to image
#14 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#14 pushing layers 1.2s done
#14 pushing manifest for docker.io/mbentley/airsonic:latest
#14 pushing manifest for docker.io/mbentley/airsonic:latest 0.3s done
#14 DONE 1.6s

#16 exporting cache
#16 sha256:2700d4ef94dee473593c5c614b55b2dedcca7893909811a8f2b48291a1f581e4
#16 preparing build cache for export done
#16 writing layer sha256:47fcdba1db8f6bb2e24cadfbf88957aecd675c89553230b5266ef1876b863ac3 done
#16 writing layer sha256:53380879c22c14c31df822b3976e5f7dc41d082e46dc33b5821183449cdd6be3 done
#16 writing layer sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b done
#16 writing layer sha256:5c126ace4b8e4d5e2d1fa6699ac82dddc270a48ac66d40ecca9bbccf7e61d697 done
#16 writing layer sha256:8057d354b41f93a2c1cf01265f97392dbd0c415c8ead5425d90a082c97db2fb7 done
#16 writing layer sha256:bfed4388b8390995898ec1f304e4cce6874d6dd620f4b84dd3b1fb0de36da27a done
#16 writing layer sha256:f74a07ff9c7f30e0525b94e45f4a04ba8130ce9fd8070cd7e8c3bc2b4e9ad810 done
#16 writing config sha256:c2a253a67a918a0511147e5777544935738f87265bb7dcbf5300d697b573b0aa 0.0s done
#16 writing manifest sha256:57e405aaf72e4bd5937ac8e80c39858b7414caec2cc2fc0e1f9d98d6606f31b8 0.0s done
#16 DONE 0.1s

Working build:

+ docker buildx build --builder builder1 --pull --push --progress plain --build-arg AIRSONIC_VER=v10.6.2 --platform linux/amd64 -t mbentley/airsonic:latest -f Dockerfile --cache-from=type=registry,ref=registry.casa.mbentley.net/mbentley/airsonic:latest-cache --cache-to=type=registry,ref=registry.casa.mbentley.net/mbentley/airsonic:latest-cache,mode=max .
#1 [internal] booting buildkit
#1 sha256:bd448caf6032bd5a5d77c0fc37ff5cecfdc320c10d188966d0df614b61366592
#1 starting container buildx_buildkit_builder1
#1 starting container buildx_buildkit_builder1 2.7s done
#1 DONE 2.7s

#2 [internal] load build definition from Dockerfile
#2 sha256:8d15bda46382c1246f234725de3207970eedeeee9d9898bb171e5645f0b778a7
#2 transferring dockerfile: 2.27kB done
#2 DONE 0.0s

#3 [internal] load .dockerignore
#3 sha256:3f4f9de50cf45693d066a337c96dfab339bf23c2f5ef8898b6736adafd224959
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [internal] load metadata for docker.io/library/alpine:latest
#4 sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6
#4 DONE 0.6s

#6 [1/8] FROM docker.io/library/alpine:latest@sha256:234cb88d3020898631af0ccbbcca9a66ae7306ecd30c9720690858c1b007d2a0
#6 sha256:83cc88488a3bbdd9d2e20ed4b0eb8ecd26a4dff43878c2e68a842fed290ad078
#6 resolve docker.io/library/alpine:latest@sha256:234cb88d3020898631af0ccbbcca9a66ae7306ecd30c9720690858c1b007d2a0 0.0s done
#6 DONE 0.0s

#5 importing cache manifest from registry.casa.mbentley.net/mbentley/airsonic:latest-cache
#5 sha256:54d6ce93e51693820c800f2f73075a6183237f347857e154552181a870bbb9d1
#5 DONE 0.0s

#12 [internal] load build context
#12 sha256:24d567b1f92090557f3d2c271a92fc28093ed628db2a310a00a55b0b94422448
#12 transferring context: 907B done
#12 DONE 0.0s

#8 [3/8] RUN (mkdir /var/airsonic &&  addgroup -g 504 airsonic &&  adduser -h /var/airsonic -D -u 504 -g airsonic -G airsonic -s /sbin/nologin airsonic &&  chown -R airsonic:airsonic /var/airsonic)
#8 sha256:1dd82e7179d413ef03ab329f8e1daf6d9d82624ebb6124f071fd8a3690381720
#8 CACHED

#7 [2/8] RUN (apk --no-cache add ca-certificates ffmpeg ttf-dejavu openjdk8 wget jq)
#7 sha256:8eaba8e869c391b7166657cc0a1455b604737370729a18d94ef373a087941341
#7 CACHED

#9 [4/8] RUN (AIRSONIC_VER="$(wget -q -O - https://api.github.com/repos/airsonic/airsonic/releases/latest | jq -r .tag_name)" &&  if [ "$(echo v10.6.2 | awk -F '.' '{print $1}')" != "v10" ]; then echo "Latest version number is no longer 10"; exit 1; fi &&  wget "https://github.com/airsonic/airsonic/releases/download/v10.6.2/airsonic.war" -O /var/airsonic/airsonic.war &&  chown airsonic:airsonic /var/airsonic/airsonic.war)
#9 sha256:3835982beb295b97e3aa4974aca74b0ba79f245ab2e64d5c62615c58cbf37051
#9 CACHED

#10 [5/8] RUN (mkdir /var/airsonic/transcode &&  ln -s /usr/bin/ffmpeg /var/airsonic/transcode/ffmpeg &&  chown -R airsonic:airsonic /var/airsonic/transcode)
#10 sha256:ad84205011c43d7cd451e5f369174eb417683b646df092e20fd67b7a4f745a5a
#10 CACHED

#11 [6/8] RUN (mkdir /data &&  cd /data &&  mkdir db index16 lucene2 lastfmcache thumbs music Podcast playlists .cache .java &&  touch airsonic.properties rollback.sql &&  cd /var/airsonic &&  ln -s /data/db &&  ln -s /data/index16 &&  ln -s /data/lucene2 &&  ln -s /data/lastfmcache &&  ln -s /data/thumbs &&  ln -s /data/music &&  ln -s /data/Podcast &&  ln -s /data/playlists &&  ln -s /data/.cache &&  ln -s /data/.java &&  ln -s /data/airsonic.properties &&  ln -s /data/rollback.sql &&  chown -R airsonic:airsonic /data)
#11 sha256:537ff7293a632ef34d77a985eca7229badd7d2b9309d5e5e11e78ce6cb790cbf
#11 CACHED

#13 [7/8] COPY entrypoint.sh /entrypoint.sh
#13 sha256:13912f5cdaad79f8c1b2301e80261308ce58310a7153e1bb02bf11d6af685ad3
#13 CACHED

#14 [8/8] WORKDIR /var/airsonic
#14 sha256:3319582f1b3b240ceb7589d54dbcdbcc712d4dfacfdff4eb8ed1d743fce52cff
#14 CACHED

#15 exporting to image
#15 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#15 exporting layers done
#15 exporting manifest sha256:29aa0bb26757325405da60b4aa7d92fa3cf0fb489982ed4c19bc3580ad94ada1 0.0s done
#15 exporting config sha256:2f71e4b4043ae8203965cb91faa008af18fcb2c15286b433c1148d36fb4d4f88 0.0s done
#15 pushing layers
#15 ...

#16 [auth] mbentley/airsonic:pull,push token for registry-1.docker.io
#16 sha256:6f923cb5824dd7970ff88871d551967c6161fbf066ccde63a0e0b8cfaa0e9ae8
#16 DONE 0.0s

#15 exporting to image
#15 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#15 pushing layers 0.7s done
#15 pushing manifest for docker.io/mbentley/airsonic:latest
#15 pushing manifest for docker.io/mbentley/airsonic:latest 0.2s done
#15 DONE 1.0s

#17 exporting cache
#17 sha256:2700d4ef94dee473593c5c614b55b2dedcca7893909811a8f2b48291a1f581e4
#17 preparing build cache for export done
#17 writing layer sha256:47fcdba1db8f6bb2e24cadfbf88957aecd675c89553230b5266ef1876b863ac3 done
#17 writing layer sha256:53380879c22c14c31df822b3976e5f7dc41d082e46dc33b5821183449cdd6be3 done
#17 writing layer sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b done
#17 writing layer sha256:5c126ace4b8e4d5e2d1fa6699ac82dddc270a48ac66d40ecca9bbccf7e61d697 done
#17 writing layer sha256:8057d354b41f93a2c1cf01265f97392dbd0c415c8ead5425d90a082c97db2fb7 done
#17 writing layer sha256:bfed4388b8390995898ec1f304e4cce6874d6dd620f4b84dd3b1fb0de36da27a done
#17 writing layer sha256:f74a07ff9c7f30e0525b94e45f4a04ba8130ce9fd8070cd7e8c3bc2b4e9ad810 done
#17 writing config sha256:c2a253a67a918a0511147e5777544935738f87265bb7dcbf5300d697b573b0aa done
#17 writing manifest sha256:57e405aaf72e4bd5937ac8e80c39858b7414caec2cc2fc0e1f9d98d6606f31b8 done
#17 DONE 0.0s
mbentley commented 3 years ago

I've been trying to track down what is going on from the logs and I believe that I have captured a build where it has problems with debug logging: builder_out.txt

time="2021-06-23T07:07:18Z" level=debug msg="do request" host=registry.casa.mbentley.net request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/manifests/4.3-armv7l-cache"
time="2021-06-23T07:07:18Z" level=debug msg=fetch digest="sha256:6aab78d1825b4c15c159fecc62b8eef4fdf0c693a15aace3a605ad44e5e2df0c" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1418
time="2021-06-23T07:07:18Z" level=debug msg="fetch response received" host=registry.casa.mbentley.net response.header.content-length=2539 response.header.content-type=application/vnd.oci.image.index.v1+json response.header.date="Wed, 23 Jun 2021 07:07:18 GMT" response.header.docker-content-digest="sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c\"" response.header.x-content-type-options=nosniff response.status="200 OK" url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/manifests/4.3-armv7l-cache"
time="2021-06-23T07:07:18Z" level=debug msg=resolved desc.digest="sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c" host=registry.casa.mbentley.net
time="2021-06-23T07:07:18Z" level=debug msg="do request" digest="sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c" request.header.accept="application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=GET url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/manifests/sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c"
time="2021-06-23T07:07:18Z" level=debug msg="fetch response received" digest="sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c" response.header.content-length=2539 response.header.content-type=application/vnd.oci.image.index.v1+json response.header.date="Wed, 23 Jun 2021 07:07:18 GMT" response.header.docker-content-digest="sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c\"" response.header.x-content-type-options=nosniff response.status="200 OK" url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/manifests/sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c"
time="2021-06-23T07:07:18Z" level=debug msg=fetch digest="sha256:f57d1d9861bfd91606b02d40f788c11da6d0156882ca29e7f69c2ccb4fbf45f0" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1150
time="2021-06-23T07:07:18Z" level=debug msg=fetch digest="sha256:df35e78d195807badda7ee058c23bcc4a4b9c75b72a13b87a6ff33d0f91fa6e3" mediatype=application/vnd.docker.container.image.v1+json size=3362
time="2021-06-23T07:07:18Z" level=debug msg="do request" digest="sha256:868ace8c6579107ce113341db658d6bc75d93c7ac687d6a21381c6000880431a" request.header.accept="application/vnd.buildkit.cacheconfig.v0, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=GET url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:868ace8c6579107ce113341db658d6bc75d93c7ac687d6a21381c6000880431a"
time="2021-06-23T07:07:18Z" level=debug msg="fetch response received" digest="sha256:868ace8c6579107ce113341db658d6bc75d93c7ac687d6a21381c6000880431a" response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=1672 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:18 GMT" response.header.docker-content-digest="sha256:868ace8c6579107ce113341db658d6bc75d93c7ac687d6a21381c6000880431a" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:868ace8c6579107ce113341db658d6bc75d93c7ac687d6a21381c6000880431a\"" response.header.x-content-type-options=nosniff response.status="200 OK" url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:868ace8c6579107ce113341db658d6bc75d93c7ac687d6a21381c6000880431a"
time="2021-06-23T07:07:18Z" level=debug msg="reusing ref for local: u9vihwqtto0662rup5cq62dd9"
time="2021-06-23T07:07:18Z" level=debug msg="diffcopy took: 2.223671ms"
time="2021-06-23T07:07:18Z" level=debug msg="saved u9vihwqtto0662rup5cq62dd9 as local.sharedKey:context:context:"
time="2021-06-23T07:07:18Z" level=debug msg="load cache for [5/5] WORKDIR /opt/tplink/EAPController/lib with sha256:2eff54abf99e05203012e344765f1ddc8b536c390e196a5574844ad72918960f"
time="2021-06-23T07:07:19Z" level=warning msg="invalid image config with unaccounted layers"
time="2021-06-23T07:07:19Z" level=debug msg=push digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=184292534
time="2021-06-23T07:07:19Z" level=debug msg=push digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json size=6173
time="2021-06-23T07:07:19Z" level=debug msg=push digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=40312433
time="2021-06-23T07:07:19Z" level=debug msg=push digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=850
time="2021-06-23T07:07:19Z" level=debug msg=push digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=949
time="2021-06-23T07:07:19Z" level=debug msg=push digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=169
time="2021-06-23T07:07:19Z" level=debug msg="checking and pushing to" digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=184292534 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2"
time="2021-06-23T07:07:19Z" level=debug msg=push digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=511
time="2021-06-23T07:07:19Z" level=debug msg="checking and pushing to" digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=949 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79"
time="2021-06-23T07:07:19Z" level=debug msg="do request" digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=949 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79"
time="2021-06-23T07:07:19Z" level=debug msg="do request" digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=184292534 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2"
time="2021-06-23T07:07:19Z" level=debug msg="checking and pushing to" digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=850 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537"
time="2021-06-23T07:07:19Z" level=debug msg="checking and pushing to" digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json size=6173 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482"
time="2021-06-23T07:07:19Z" level=debug msg="do request" digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=850 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537"
time="2021-06-23T07:07:19Z" level=debug msg="checking and pushing to" digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=511 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e"
time="2021-06-23T07:07:19Z" level=debug msg="checking and pushing to" digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=169 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73"
time="2021-06-23T07:07:19Z" level=debug msg="do request" digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=511 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e"
time="2021-06-23T07:07:19Z" level=debug msg="checking and pushing to" digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=40312433 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a"
time="2021-06-23T07:07:19Z" level=debug msg="do request" digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=6173 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482"
time="2021-06-23T07:07:19Z" level=debug msg="do request" digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=169 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73"
time="2021-06-23T07:07:19Z" level=debug msg="do request" digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=40312433 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a"
time="2021-06-23T07:07:19Z" level=debug msg="fetch response received" digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=12792 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0f479a417ff-ATL response.header.cf-request-id=0ad94aecce000017ffc922c000000001 response.header.connection=keep-alive response.header.content-length=949 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:19 GMT" response.header.etag="\"ae9096e54497bc950017802d8c433ced\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:19 GMT" response.header.last-modified="Tue, 08 Jun 2021 22:22:19 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="7WHMOgjQWBej88N+J8qCLME+OvgPkVizQHt0KhAsT6ChLPJCNmBsEl0mZd8J+5UCbjCMuV/PTfg=" response.header.x-amz-request-id=T2KB2RFNQGKJRKT0 response.header.x-amz-version-id=A0G.e_QwxTR4DpkYEexuYxx2uhrDAtsv response.status="200 OK" size=949 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79"
time="2021-06-23T07:07:19Z" level=debug msg="fetch response received" digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=184317 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0f4ca4717ff-ATL response.header.cf-request-id=0ad94aecfa000017ff86137000000001 response.header.connection=keep-alive response.header.content-length=511 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:19 GMT" response.header.etag="\"c9c445daf73d19cd9e2f49a46e979847\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:19 GMT" response.header.last-modified="Thu, 17 Jun 2021 23:36:34 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="Lu4EwPGfBYg9XZ3Gcz+l+flwUdioFuYVMhZ+izXOe8VvN8Jl5KcZlsVPRS/CGhYJJYKpUpVJbvk=" response.header.x-amz-request-id=5Q23WV5ETC9JJQEJ response.header.x-amz-version-id=kr100x6ciJ.v_gdcQV6zoIxoE_jv_5Yb response.status="200 OK" size=511 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e"
time="2021-06-23T07:07:19Z" level=debug msg="fetch response received" digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json response.header.content-length=157 response.header.content-type=application/json response.header.date="Wed, 23 Jun 2021 07:07:19 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.strict-transport-security="max-age=31536000" response.status="404 Not Found" size=6173 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482"
time="2021-06-23T07:07:19Z" level=debug msg="do request" digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json request.header.user-agent=containerd/1.5.2+unknown request.method=POST size=6173 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/uploads/"
time="2021-06-23T07:07:19Z" level=debug msg="fetch response received" digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=18560 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0f51adc17ff-ATL response.header.cf-request-id=0ad94aed2f000017ffb1b7e000000001 response.header.connection=keep-alive response.header.content-length=850 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:19 GMT" response.header.etag="\"4f6d27829a5a8848dc4a345f16584a9d\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:19 GMT" response.header.last-modified="Thu, 17 Jun 2021 23:36:34 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="zbW3msFftJaH0X5HG612RVtW6iCJZ74OkAS6ADAZGFADvEeeVBFXTU8pxqZR6i0L4FevsmpyHXY=" response.header.x-amz-request-id=5Q20PG1ZAN3KEE7F response.header.x-amz-version-id=jzwOX7X.RDtmxFGniUjS9KcuwusAfpnf response.status="200 OK" size=850 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537"
time="2021-06-23T07:07:19Z" level=debug msg="fetch response received" digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0f47efaf339-ATL response.header.cf-request-id=0ad94aeccc0000f339e4b62000000001 response.header.connection=keep-alive response.header.content-length=184292534 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:19 GMT" response.header.etag="\"8cffc52c9abbe367a0ab1a767e86646d-6\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:19 GMT" response.header.last-modified="Tue, 22 Jun 2021 02:51:43 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="STL6mrG8aI64INF/k7s6fngTjQnIuZKYAZiqEyXGAwJV564a2GPWXtI1s0ndIooVOhE4TRBNTY4=" response.header.x-amz-request-id=N06SPJEQQSVF8MDH response.header.x-amz-version-id=VKsONpj1xB0iUgFo86Gk9oAbELTQ7I8j response.status="200 OK" size=184292534 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2"
time="2021-06-23T07:07:19Z" level=debug msg="fetch response received" digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=184317 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0f54b3617ff-ATL response.header.cf-request-id=0ad94aed50000017ffec8ce000000001 response.header.connection=keep-alive response.header.content-length=40312433 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:19 GMT" response.header.etag="\"6988f4beb12987d787991dda34637048-2\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:19 GMT" response.header.last-modified="Thu, 17 Jun 2021 23:36:42 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="16MBeRly3yltgfcv42Z+bTGtIxiBwZ5kMm5MaxFmvH92Zc/tAdPu5qUJxCBZ2heqRJq4Q52nE7c=" response.header.x-amz-request-id=CC85WV630G266XWQ response.header.x-amz-version-id=v_D4kFqDj04FLMiA0Ek52VX64v7_rpHk response.status="200 OK" size=40312433 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a"
time="2021-06-23T07:07:19Z" level=debug msg="fetch response received" digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=184312 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0f56a021035-ATL response.header.cf-request-id=0ad94aed5d00001035bfa24000000001 response.header.connection=keep-alive response.header.content-length=169 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:19 GMT" response.header.etag="\"1e8687c8dd0c8f6a82515154f0a87757\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:19 GMT" response.header.last-modified="Thu, 17 Jun 2021 23:36:34 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="vVudmtZ2sCQfOmVaPVk0D8D95tVsju9ZIY45v1NcGcodUn8Oj6aiDaUE+Io6yNAN3WM/HOP+RDk=" response.header.x-amz-request-id=5Q270MYTS1PQB3MQ response.header.x-amz-version-id=TBf6wQhJNo9p71SbmGiB4VApVsgguVzv response.status="200 OK" size=169 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73"
time="2021-06-23T07:07:19Z" level=debug msg="fetch response received" digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json response.header.content-length=0 response.header.date="Wed, 23 Jun 2021 07:07:19 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-upload-uuid=da6c5210-a93a-4ab1-a68a-dd375743e65e response.header.location="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/uploads/da6c5210-a93a-4ab1-a68a-dd375743e65e?_state=04cH2mRVJDE7XJial7H04GhdXXUAnxXbJp7qrvHoIcp7Ik5hbWUiOiJtYmVudGxleS9vbWFkYS1jb250cm9sbGVyIiwiVVVJRCI6ImRhNmM1MjEwLWE5M2EtNGFiMS1hNjhhLWRkMzc1NzQzZTY1ZSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMS0wNi0yM1QwNzowNzoxOS4yMzExNTM4NTJaIn0%3D" response.header.range=0-0 response.header.strict-transport-security="max-age=31536000" response.status="202 Accepted" size=6173 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/uploads/"
time="2021-06-23T07:07:19Z" level=debug msg="do request" digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json request.header.content-type=application/octet-stream request.header.user-agent=containerd/1.5.2+unknown request.method=PUT size=6173 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/uploads/da6c5210-a93a-4ab1-a68a-dd375743e65e?_state=04cH2mRVJDE7XJial7H04GhdXXUAnxXbJp7qrvHoIcp7Ik5hbWUiOiJtYmVudGxleS9vbWFkYS1jb250cm9sbGVyIiwiVVVJRCI6ImRhNmM1MjEwLWE5M2EtNGFiMS1hNjhhLWRkMzc1NzQzZTY1ZSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMS0wNi0yM1QwNzowNzoxOS4yMzExNTM4NTJaIn0%3D&digest=sha256%3A0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482"
time="2021-06-23T07:07:19Z" level=debug msg="fetch response received" digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json response.header.content-length=0 response.header.date="Wed, 23 Jun 2021 07:07:19 GMT" response.header.docker-content-digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" response.header.docker-distribution-api-version=registry/2.0 response.header.location="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" response.header.strict-transport-security="max-age=31536000" response.status="201 Created" size=6173 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/uploads/da6c5210-a93a-4ab1-a68a-dd375743e65e?_state=04cH2mRVJDE7XJial7H04GhdXXUAnxXbJp7qrvHoIcp7Ik5hbWUiOiJtYmVudGxleS9vbWFkYS1jb250cm9sbGVyIiwiVVVJRCI6ImRhNmM1MjEwLWE5M2EtNGFiMS1hNjhhLWRkMzc1NzQzZTY1ZSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMS0wNi0yM1QwNzowNzoxOS4yMzExNTM4NTJaIn0%3D&digest=sha256%3A0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482"
time="2021-06-23T07:07:19Z" level=debug msg=push digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1570
time="2021-06-23T07:07:19Z" level=debug msg="checking and pushing to" digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1570 url="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/4.3-armv7l"
time="2021-06-23T07:07:19Z" level=debug msg="do request" digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=1570 url="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/4.3-armv7l"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=1778 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.docker-content-digest="sha256:32dcc37839232302adca5f25f31476253bd58415080da89a31c535d7f1101b3e" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:32dcc37839232302adca5f25f31476253bd58415080da89a31c535d7f1101b3e\"" response.header.ratelimit-limit="200;w=21600" response.header.ratelimit-remaining="195;w=21600" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" size=1570 url="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/4.3-armv7l"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.content-type=application/vnd.docker.distribution.manifest.v2+json request.header.user-agent=containerd/1.5.2+unknown request.method=PUT size=1570 url="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/4.3-armv7l"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=0 response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.docker-content-digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" response.header.docker-distribution-api-version=registry/2.0 response.header.location="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" response.header.strict-transport-security="max-age=31536000" response.status="201 Created" size=1570 url="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/4.3-armv7l"
time="2021-06-23T07:07:20Z" level=debug msg=push digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=184292534
time="2021-06-23T07:07:20Z" level=debug msg=push digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=511
time="2021-06-23T07:07:20Z" level=debug msg=push digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=850
time="2021-06-23T07:07:20Z" level=debug msg=push digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=949
time="2021-06-23T07:07:20Z" level=debug msg=push digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=169
time="2021-06-23T07:07:20Z" level=debug msg=push digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json size=6173
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=184292534 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2"
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=850 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537"
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json size=6173 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=184292534 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2"
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=511 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=6173 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482"
time="2021-06-23T07:07:20Z" level=debug msg=push digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=40312433
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=850 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537"
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=169 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73"
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=949 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=511 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e"
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=40312433 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=949 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=40312433 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=169 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=184318 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0fbadbd17ff-ATL response.header.cf-request-id=0ad94af14d000017ffca16a000000001 response.header.connection=keep-alive response.header.content-length=511 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.etag="\"c9c445daf73d19cd9e2f49a46e979847\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:20 GMT" response.header.last-modified="Thu, 17 Jun 2021 23:36:34 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="Lu4EwPGfBYg9XZ3Gcz+l+flwUdioFuYVMhZ+izXOe8VvN8Jl5KcZlsVPRS/CGhYJJYKpUpVJbvk=" response.header.x-amz-request-id=5Q23WV5ETC9JJQEJ response.header.x-amz-version-id=kr100x6ciJ.v_gdcQV6zoIxoE_jv_5Yb response.status="200 OK" size=511 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=1 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0fbbf90f339-ATL response.header.cf-request-id=0ad94af1530000f3392ba04000000001 response.header.connection=keep-alive response.header.content-length=184292534 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.etag="\"8cffc52c9abbe367a0ab1a767e86646d-6\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:20 GMT" response.header.last-modified="Tue, 22 Jun 2021 02:51:43 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="STL6mrG8aI64INF/k7s6fngTjQnIuZKYAZiqEyXGAwJV564a2GPWXtI1s0ndIooVOhE4TRBNTY4=" response.header.x-amz-request-id=N06SPJEQQSVF8MDH response.header.x-amz-version-id=VKsONpj1xB0iUgFo86Gk9oAbELTQ7I8j response.status="200 OK" size=184292534 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=184318 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0fbfe2f17ff-ATL response.header.cf-request-id=0ad94af178000017ffbc3a9000000001 response.header.connection=keep-alive response.header.content-length=40312433 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.etag="\"6988f4beb12987d787991dda34637048-2\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:20 GMT" response.header.last-modified="Thu, 17 Jun 2021 23:36:42 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="16MBeRly3yltgfcv42Z+bTGtIxiBwZ5kMm5MaxFmvH92Zc/tAdPu5qUJxCBZ2heqRJq4Q52nE7c=" response.header.x-amz-request-id=CC85WV630G266XWQ response.header.x-amz-version-id=v_D4kFqDj04FLMiA0Ek52VX64v7_rpHk response.status="200 OK" size=40312433 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=18561 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0fc3ea417ff-ATL response.header.cf-request-id=0ad94af1a4000017ffa50e1000000001 response.header.connection=keep-alive response.header.content-length=850 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.etag="\"4f6d27829a5a8848dc4a345f16584a9d\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:20 GMT" response.header.last-modified="Thu, 17 Jun 2021 23:36:34 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="zbW3msFftJaH0X5HG612RVtW6iCJZ74OkAS6ADAZGFADvEeeVBFXTU8pxqZR6i0L4FevsmpyHXY=" response.header.x-amz-request-id=5Q20PG1ZAN3KEE7F response.header.x-amz-version-id=jzwOX7X.RDtmxFGniUjS9KcuwusAfpnf response.status="200 OK" size=850 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=12793 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0fc6f0e17ff-ATL response.header.cf-request-id=0ad94af1c5000017ff991c3000000001 response.header.connection=keep-alive response.header.content-length=949 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.etag="\"ae9096e54497bc950017802d8c433ced\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:20 GMT" response.header.last-modified="Tue, 08 Jun 2021 22:22:19 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="7WHMOgjQWBej88N+J8qCLME+OvgPkVizQHt0KhAsT6ChLPJCNmBsEl0mZd8J+5UCbjCMuV/PTfg=" response.header.x-amz-request-id=T2KB2RFNQGKJRKT0 response.header.x-amz-version-id=A0G.e_QwxTR4DpkYEexuYxx2uhrDAtsv response.status="200 OK" size=949 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=184313 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=663be0fcbf8717ff-ATL response.header.cf-request-id=0ad94af1f0000017ffb801e000000001 response.header.connection=keep-alive response.header.content-length=169 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.etag="\"1e8687c8dd0c8f6a82515154f0a87757\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:20 GMT" response.header.last-modified="Thu, 17 Jun 2021 23:36:34 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="vVudmtZ2sCQfOmVaPVk0D8D95tVsju9ZIY45v1NcGcodUn8Oj6aiDaUE+Io6yNAN3WM/HOP+RDk=" response.header.x-amz-request-id=5Q270MYTS1PQB3MQ response.header.x-amz-version-id=TBf6wQhJNo9p71SbmGiB4VApVsgguVzv response.status="200 OK" size=169 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=MISS response.header.cf-ray=663be0fc0feff339-ATL response.header.cf-request-id=0ad94af1850000f339058be000000001 response.header.connection=keep-alive response.header.content-length=6173 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.etag="\"59e2b30f2b80a4593550cbd574f74a31\"" response.header.expect-ct="max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"" response.header.expires="Wed, 23 Jun 2021 11:07:20 GMT" response.header.last-modified="Wed, 23 Jun 2021 07:07:20 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="RQttWkdAslqzicVJKdaHHK/0wMnmvUsIr2C9ubRmczTUOkwYN6Rpwrt0ouXHBZOnc2ZddOhlBeI=" response.header.x-amz-request-id=8FXJKKVNKNPHCKF1 response.header.x-amz-version-id=KC4e.PaWd0SSmjPe4JHy9EC86cPZCDin response.status="200 OK" size=6173 url="https://registry-1.docker.io/v2/mbentley/omada-controller/blobs/sha256:0a09e2786ba6dde308248059eddcab43ce7cbd382f303621b34ea3d732d42482"
time="2021-06-23T07:07:20Z" level=debug msg=push digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1570
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1570 url="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/latest-armv7l"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=1570 url="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/latest-armv7l"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=1778 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.docker-content-digest="sha256:32dcc37839232302adca5f25f31476253bd58415080da89a31c535d7f1101b3e" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:32dcc37839232302adca5f25f31476253bd58415080da89a31c535d7f1101b3e\"" response.header.ratelimit-limit="200;w=21600" response.header.ratelimit-remaining="195;w=21600" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" size=1570 url="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/latest-armv7l"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.content-type=application/vnd.docker.distribution.manifest.v2+json request.header.user-agent=containerd/1.5.2+unknown request.method=PUT size=1570 url="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/latest-armv7l"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=0 response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.docker-content-digest="sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" response.header.docker-distribution-api-version=registry/2.0 response.header.location="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d" response.header.strict-transport-security="max-age=31536000" response.status="201 Created" size=1570 url="https://registry-1.docker.io/v2/mbentley/omada-controller/manifests/latest-armv7l"
time="2021-06-23T07:07:20Z" level=debug msg=fetch digest="sha256:4cf1ff4fc0c6d1d91beb4f3889f10c2a95b0afb3a9de12c877de952155627c0f" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=1918
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:4cf1ff4fc0c6d1d91beb4f3889f10c2a95b0afb3a9de12c877de952155627c0f" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=1918 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:4cf1ff4fc0c6d1d91beb4f3889f10c2a95b0afb3a9de12c877de952155627c0f"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:4cf1ff4fc0c6d1d91beb4f3889f10c2a95b0afb3a9de12c877de952155627c0f" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=1918 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:4cf1ff4fc0c6d1d91beb4f3889f10c2a95b0afb3a9de12c877de952155627c0f"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:4cf1ff4fc0c6d1d91beb4f3889f10c2a95b0afb3a9de12c877de952155627c0f" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=1918 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.docker-content-digest="sha256:4cf1ff4fc0c6d1d91beb4f3889f10c2a95b0afb3a9de12c877de952155627c0f" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:4cf1ff4fc0c6d1d91beb4f3889f10c2a95b0afb3a9de12c877de952155627c0f\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=1918 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:4cf1ff4fc0c6d1d91beb4f3889f10c2a95b0afb3a9de12c877de952155627c0f"
time="2021-06-23T07:07:20Z" level=debug msg=fetch digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=850
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=850 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=850 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=850 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.docker-content-digest="sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=850 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:54f4e7436b523284825c35db6cb48a7c20c8d5388815695277126f528b2b4537"
time="2021-06-23T07:07:20Z" level=debug msg=fetch digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=184292534
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=184292534 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=184292534 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2"
time="2021-06-23T07:07:20Z" level=debug msg="fetch response received" digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=184292534 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:20 GMT" response.header.docker-content-digest="sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=184292534 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:5c2ee5fefda0565d952161aaff0c24e2b945d4e620db9daf61d8473f99eebfe2"
time="2021-06-23T07:07:20Z" level=debug msg=fetch digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=949
time="2021-06-23T07:07:20Z" level=debug msg="checking and pushing to" digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=949 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79"
time="2021-06-23T07:07:20Z" level=debug msg="do request" digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=949 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79"
time="2021-06-23T07:07:21Z" level=debug msg="fetch response received" digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=949 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:21 GMT" response.header.docker-content-digest="sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=949 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:859688b1f4a3b55fb268999ac651fa105675adee85268db5ed865235f547cc79"
time="2021-06-23T07:07:21Z" level=debug msg=fetch digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=511
time="2021-06-23T07:07:21Z" level=debug msg="checking and pushing to" digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=511 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e"
time="2021-06-23T07:07:21Z" level=debug msg="do request" digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=511 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e"
time="2021-06-23T07:07:21Z" level=debug msg="fetch response received" digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=511 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:21 GMT" response.header.docker-content-digest="sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=511 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:ad5c9e5711d0203aa41ec7750be25bd3bdd00b90388fde5a0d603523354eff4e"
time="2021-06-23T07:07:21Z" level=debug msg=fetch digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=40312433
time="2021-06-23T07:07:21Z" level=debug msg="checking and pushing to" digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=40312433 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a"
time="2021-06-23T07:07:21Z" level=debug msg="do request" digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=40312433 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a"
time="2021-06-23T07:07:21Z" level=debug msg="fetch response received" digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=40312433 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:21 GMT" response.header.docker-content-digest="sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=40312433 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:bee35c084ef2789f844b87c031d55b07186255622024a15b526838f71500485a"
time="2021-06-23T07:07:21Z" level=debug msg=fetch digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=169
time="2021-06-23T07:07:21Z" level=debug msg="checking and pushing to" digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=169 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73"
time="2021-06-23T07:07:21Z" level=debug msg="do request" digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD size=169 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73"
time="2021-06-23T07:07:21Z" level=debug msg="fetch response received" digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=169 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:21 GMT" response.header.docker-content-digest="sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=169 url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:c0d1eba314dc97cbd3e70e1b857a42881405eefe61a235d56722dbf0df809d73"
time="2021-06-23T07:07:21Z" level=warning msg="reference for unknown type: application/vnd.buildkit.cacheconfig.v0"
time="2021-06-23T07:07:21Z" level=debug msg="checking and pushing to" url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:265a7fbd257218403004ddfa8388192d56395aca85b976a0bdffc99422654ef0"
time="2021-06-23T07:07:21Z" level=debug msg="do request" request.header.accept="application/vnd.buildkit.cacheconfig.v0, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:265a7fbd257218403004ddfa8388192d56395aca85b976a0bdffc99422654ef0"
time="2021-06-23T07:07:21Z" level=debug msg="fetch response received" response.header.content-length=157 response.header.content-type="application/json; charset=utf-8" response.header.date="Wed, 23 Jun 2021 07:07:21 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.x-content-type-options=nosniff response.status="404 Not Found" url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:265a7fbd257218403004ddfa8388192d56395aca85b976a0bdffc99422654ef0"
time="2021-06-23T07:07:21Z" level=debug msg="do request" request.header.user-agent=containerd/1.5.2+unknown request.method=POST url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/uploads/"
time="2021-06-23T07:07:21Z" level=debug msg="fetch response received" response.header.content-length=0 response.header.date="Wed, 23 Jun 2021 07:07:21 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-upload-uuid=9597a4a6-c057-4cc9-b320-755c45120e73 response.header.location="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/uploads/9597a4a6-c057-4cc9-b320-755c45120e73?_state=3N_LcTGS3T4gKATojBeGAXjHb5UuWYAhERB4uUVArMR7Ik5hbWUiOiJtYmVudGxleS9vbWFkYS1jb250cm9sbGVyIiwiVVVJRCI6Ijk1OTdhNGE2LWMwNTctNGNjOS1iMzIwLTc1NWM0NTEyMGU3MyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMS0wNi0yM1QwNzowNzoyMS4wMTYwMTkyMjNaIn0%3D" response.header.range=0-0 response.header.x-content-type-options=nosniff response.status="202 Accepted" url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/uploads/"
time="2021-06-23T07:07:21Z" level=debug msg="do request" request.header.content-type=application/octet-stream request.header.user-agent=containerd/1.5.2+unknown request.method=PUT url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/uploads/9597a4a6-c057-4cc9-b320-755c45120e73?_state=3N_LcTGS3T4gKATojBeGAXjHb5UuWYAhERB4uUVArMR7Ik5hbWUiOiJtYmVudGxleS9vbWFkYS1jb250cm9sbGVyIiwiVVVJRCI6Ijk1OTdhNGE2LWMwNTctNGNjOS1iMzIwLTc1NWM0NTEyMGU3MyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMS0wNi0yM1QwNzowNzoyMS4wMTYwMTkyMjNaIn0%3D&digest=sha256%3A265a7fbd257218403004ddfa8388192d56395aca85b976a0bdffc99422654ef0"
time="2021-06-23T07:07:21Z" level=debug msg="fetch response received" response.header.content-length=0 response.header.date="Wed, 23 Jun 2021 07:07:21 GMT" response.header.docker-content-digest="sha256:265a7fbd257218403004ddfa8388192d56395aca85b976a0bdffc99422654ef0" response.header.docker-distribution-api-version=registry/2.0 response.header.location="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/sha256:265a7fbd257218403004ddfa8388192d56395aca85b976a0bdffc99422654ef0" response.header.x-content-type-options=nosniff response.status="201 Created" url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/blobs/uploads/9597a4a6-c057-4cc9-b320-755c45120e73?_state=3N_LcTGS3T4gKATojBeGAXjHb5UuWYAhERB4uUVArMR7Ik5hbWUiOiJtYmVudGxleS9vbWFkYS1jb250cm9sbGVyIiwiVVVJRCI6Ijk1OTdhNGE2LWMwNTctNGNjOS1iMzIwLTc1NWM0NTEyMGU3MyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMS0wNi0yM1QwNzowNzoyMS4wMTYwMTkyMjNaIn0%3D&digest=sha256%3A265a7fbd257218403004ddfa8388192d56395aca85b976a0bdffc99422654ef0"
time="2021-06-23T07:07:21Z" level=debug msg="checking and pushing to" url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/manifests/4.3-armv7l-cache"
time="2021-06-23T07:07:21Z" level=debug msg="do request" request.header.accept="application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/manifests/4.3-armv7l-cache"
time="2021-06-23T07:07:21Z" level=debug msg="fetch response received" response.header.content-length=2539 response.header.content-type=application/vnd.oci.image.index.v1+json response.header.date="Wed, 23 Jun 2021 07:07:21 GMT" response.header.docker-content-digest="sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a4233b995351edf05e7c6efb3ceee0b491a49a7d4f642c75b21d6dd8cf3e637c\"" response.header.x-content-type-options=nosniff response.status="200 OK" url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/manifests/4.3-armv7l-cache"
time="2021-06-23T07:07:21Z" level=debug msg="do request" request.header.content-type=application/vnd.oci.image.index.v1+json request.header.user-agent=containerd/1.5.2+unknown request.method=PUT url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/manifests/4.3-armv7l-cache"
time="2021-06-23T07:07:21Z" level=debug msg="fetch response received" response.header.content-length=0 response.header.date="Wed, 23 Jun 2021 07:07:21 GMT" response.header.docker-content-digest="sha256:e09dfbc81c3a48c492a933d2e9a9833d96b200b405b6ad1ab063e82e2e7dafd5" response.header.docker-distribution-api-version=registry/2.0 response.header.location="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/manifests/sha256:e09dfbc81c3a48c492a933d2e9a9833d96b200b405b6ad1ab063e82e2e7dafd5" response.header.x-content-type-options=nosniff response.status="201 Created" url="https://registry.casa.mbentley.net/v2/mbentley/omada-controller/manifests/4.3-armv7l-cache"
time="2021-06-23T07:07:21Z" level=debug msg="session finished: <nil>"
time="2021-06-23T07:07:21Z" level=debug msg="session started"
time="2021-06-23T07:07:22Z" level=debug msg="reusing ref for local: 6q55sn38mts9hsi2z5ziqhxs0"
time="2021-06-23T07:07:22Z" level=debug msg="diffcopy took: 4.526028ms"
time="2021-06-23T07:07:22Z" level=debug msg="saved 6q55sn38mts9hsi2z5ziqhxs0 as local.sharedKey:dockerfile:dockerfile:"
time="2021-06-23T07:07:22Z" level=debug msg="reusing ref for local: n9sy7qi4kc4ty4hlanij8nb29"
time="2021-06-23T07:07:22Z" level=debug msg="diffcopy took: 3.354888ms"
time="2021-06-23T07:07:22Z" level=debug msg="saved n9sy7qi4kc4ty4hlanij8nb29 as local.sharedKey:context:context-.dockerignore:"
time="2021-06-23T07:07:22Z" level=debug msg=resolving host=registry-mirror.casa.mbentley.net
time="2021-06-23T07:07:22Z" level=debug msg="do request" host=registry-mirror.casa.mbentley.net request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=HEAD url="https://registry-mirror.casa.mbentley.net/v2/library/ubuntu/manifests/18.04?ns=docker.io"
time="2021-06-23T07:07:22Z" level=debug msg="fetch response received" host=registry-mirror.casa.mbentley.net response.header.content-length=1412 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Wed, 23 Jun 2021 07:07:22 GMT" response.header.docker-content-digest="sha256:139b3846cee2e63de9ced83cee7023a2d95763ee2573e5b0ab6dea9dfbd4db8f" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:139b3846cee2e63de9ced83cee7023a2d95763ee2573e5b0ab6dea9dfbd4db8f\"" response.header.x-content-type-options=nosniff response.status="200 OK" url="https://registry-mirror.casa.mbentley.net/v2/library/ubuntu/manifests/18.04?ns=docker.io"
time="2021-06-23T07:07:22Z" level=debug msg=resolved desc.digest="sha256:139b3846cee2e63de9ced83cee7023a2d95763ee2573e5b0ab6dea9dfbd4db8f" host=registry-mirror.casa.mbentley.net
time="2021-06-23T07:07:22Z" level=debug msg=fetch digest="sha256:139b3846cee2e63de9ced83cee7023a2d95763ee2573e5b0ab6dea9dfbd4db8f" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1412
time="2021-06-23T07:07:22Z" level=debug msg=fetch digest="sha256:c60266b67f58fafc30703315f617a8fcccaffc48ef5534ca5f67a9ba3aceb3b8" mediatype=application/vnd.docker.distribution.manifest.v2+json size=529
time="2021-06-23T07:07:22Z" level=debug msg=fetch digest="sha256:d2bb498b40f3c40cb11963f63e948c15222c43587b35296389b67e3cbbcd2bb7" mediatype=application/vnd.docker.distribution.manifest.v2+json size=529
time="2021-06-23T07:07:22Z" level=debug msg=fetch digest="sha256:86d265393168783194e0a2d0e43b937bfd3f42fe10da20a90ac08656db453995" mediatype=application/vnd.docker.container.image.v1+json size=1463
time="2021-06-23T07:07:22Z" level=debug msg="do request" digest="sha256:d2bb498b40f3c40cb11963f63e948c15222c43587b35296389b67e3cbbcd2bb7" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=GET size=529 url="https://registry-mirror.casa.mbentley.net/v2/library/ubuntu/manifests/sha256:d2bb498b40f3c40cb11963f63e948c15222c43587b35296389b67e3cbbcd2bb7?ns=docker.io"
time="2021-06-23T07:07:22Z" level=debug msg="fetch response received" digest="sha256:d2bb498b40f3c40cb11963f63e948c15222c43587b35296389b67e3cbbcd2bb7" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=529 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 23 Jun 2021 07:07:22 GMT" response.header.docker-content-digest="sha256:d2bb498b40f3c40cb11963f63e948c15222c43587b35296389b67e3cbbcd2bb7" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:d2bb498b40f3c40cb11963f63e948c15222c43587b35296389b67e3cbbcd2bb7\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=529 url="https://registry-mirror.casa.mbentley.net/v2/library/ubuntu/manifests/sha256:d2bb498b40f3c40cb11963f63e948c15222c43587b35296389b67e3cbbcd2bb7?ns=docker.io"
time="2021-06-23T07:07:22Z" level=debug msg=fetch digest="sha256:b5ceeb7128a1ec71ed86e1b0da63ca87456824458e67ca4f688a12c061964241" mediatype=application/vnd.docker.container.image.v1+json size=1461
time="2021-06-23T07:07:22Z" level=debug msg="do request" digest="sha256:b5ceeb7128a1ec71ed86e1b0da63ca87456824458e67ca4f688a12c061964241" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.2+unknown request.method=GET size=1461 url="https://registry-mirror.casa.mbentley.net/v2/library/ubuntu/blobs/sha256:b5ceeb7128a1ec71ed86e1b0da63ca87456824458e67ca4f688a12c061964241?ns=docker.io"
time="2021-06-23T07:07:22Z" level=debug msg="fetch response received" digest="sha256:b5ceeb7128a1ec71ed86e1b0da63ca87456824458e67ca4f688a12c061964241" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=1461 response.header.content-type=application/octet-stream response.header.date="Wed, 23 Jun 2021 07:07:22 GMT" response.header.docker-content-digest="sha256:b5ceeb7128a1ec71ed86e1b0da63ca87456824458e67ca4f688a12c061964241" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:b5ceeb7128a1ec71ed86e1b0da63ca87456824458e67ca4f688a12c061964241\"" response.header.x-content-type-options=nosniff response.status="200 OK" size=1461 url="https://registry-mirror.casa.mbentley.net/v2/library/ubuntu/blobs/sha256:b5ceeb7128a1ec71ed86e1b0da63ca87456824458e67ca4f688a12c061964241?ns=docker.io"
time="2021-06-23T07:07:22Z" level=debug msg=resolving host=registry.casa.mbentley.net

It seems like every build I do, I get the "reference for unknown type: application/vnd.buildkit.cacheconfig.v0" warning and occasionally "invalid image config with unaccounted layers" but I am not sure that specifically has anything to do with the problem I am seeing. Looking at the above log output, I am not sure I see any smoking gun either so I am not exactly sure what's happening. I do know that the image mbentley/omada-controller:4.3-armv7l@sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d does show as an image that has the same issue though as copying over the entrypoint shouldn't be 0B:

$ docker history mbentley/omada-controller:4.3-armv7l@sha256:0bce605f09a20b0d702754c22aeb3f99438db09482fbe5b2fb697230d0cabe6d
IMAGE          CREATED      CREATED BY                                      SIZE      COMMENT
0a09e2786ba6   2 days ago   CMD ["/usr/bin/java" "-server" "-Xms128m" "-…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   ENTRYPOINT ["/entrypoint.sh"]                   0B        buildkit.dockerfile.v0
<missing>      2 days ago   VOLUME [/opt/tplink/EAPController/data /opt/…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   HEALTHCHECK &{["CMD-SHELL" "/healthcheck.sh"…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   EXPOSE map[27001/udp:{} 27002/tcp:{} 29810/u…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   WORKDIR /opt/tplink/EAPController/lib           0B        buildkit.dockerfile.v0
<missing>      2 days ago   COPY entrypoint-4.x.sh /entrypoint.sh # buil…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   RUN |4 OMADA_VER=4.3.5 OMADA_TAR=Omada_SDN_C…   315MB     buildkit.dockerfile.v0
<missing>      2 days ago   COPY install.sh healthcheck.sh / # buildkit     1.72kB    buildkit.dockerfile.v0
<missing>      2 days ago   ARG ARCH=amd64                                  0B        buildkit.dockerfile.v0
<missing>      2 days ago   ARG OMADA_URL=https://static.tp-link.com/202…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   ARG OMADA_TAR=Omada_SDN_Controller_v4.3.5_li…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   ARG OMADA_VER=4.3.5                             0B        buildkit.dockerfile.v0
<missing>      2 days ago   LABEL maintainer=Matt Bentley <mbentley@mben…   0B        buildkit.dockerfile.v0
<missing>      6 days ago   /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>      6 days ago   /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B
<missing>      6 days ago   /bin/sh -c rm -rf /var/lib/apt/lists/*          0B
<missing>      6 days ago   /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   745B
<missing>      6 days ago   /bin/sh -c #(nop) ADD file:ef24ce1c15acdd071…   106MB

What it should be:

$ docker history mbentley/omada-controller:4.3-armv7l@sha256:32dcc37839232302adca5f25f31476253bd58415080da89a31c535d7f1101b3e
IMAGE          CREATED      CREATED BY                                      SIZE      COMMENT
eee41e1f94f9   2 days ago   CMD ["/usr/bin/java" "-server" "-Xms128m" "-…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   ENTRYPOINT ["/entrypoint.sh"]                   0B        buildkit.dockerfile.v0
<missing>      2 days ago   VOLUME [/opt/tplink/EAPController/data /opt/…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   HEALTHCHECK &{["CMD-SHELL" "/healthcheck.sh"…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   EXPOSE map[27001/udp:{} 27002/tcp:{} 29810/u…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   WORKDIR /opt/tplink/EAPController/lib           0B        buildkit.dockerfile.v0
<missing>      2 days ago   COPY entrypoint-4.x.sh /entrypoint.sh # buil…   4.97kB    buildkit.dockerfile.v0
<missing>      2 days ago   RUN |4 OMADA_VER=4.3.5 OMADA_TAR=Omada_SDN_C…   315MB     buildkit.dockerfile.v0
<missing>      2 days ago   COPY install.sh healthcheck.sh / # buildkit     1.72kB    buildkit.dockerfile.v0
<missing>      2 days ago   ARG ARCH=amd64                                  0B        buildkit.dockerfile.v0
<missing>      2 days ago   ARG OMADA_URL=https://static.tp-link.com/202…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   ARG OMADA_TAR=Omada_SDN_Controller_v4.3.5_li…   0B        buildkit.dockerfile.v0
<missing>      2 days ago   ARG OMADA_VER=4.3.5                             0B        buildkit.dockerfile.v0
<missing>      2 days ago   LABEL maintainer=Matt Bentley <mbentley@mben…   0B        buildkit.dockerfile.v0
<missing>      6 days ago   /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>      6 days ago   /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B
<missing>      6 days ago   /bin/sh -c rm -rf /var/lib/apt/lists/*          0B
<missing>      6 days ago   /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   745B
<missing>      6 days ago   /bin/sh -c #(nop) ADD file:ef24ce1c15acdd071…   106MB
tonistiigi commented 3 years ago

Do you have a runnable reproducer. I see you have a repo with code but don't see the exact order of commands that would reproduce this.

mbentley commented 3 years ago

Thanks for the reply. I was spending this morning trying to get a way to reproduce this and have not yet figured out how to do so. It doesn't seem to have anything to do with specific commands or files in a build as I've seen it happen with different builds of two completely different applications and different layers. I've seen it happen with both RUN and COPY layers.

I'll keep trying to see what I might be able to do to reproduce this. I was also curious if there was anything else I should be doing to try to capture the right information. I've added --debug to the buildkit flags but I wasn't sure if there was anything else I should be doing to make sure I can capture any relevant info.

tonistiigi commented 3 years ago

My hunch is that this has something to do with RUN commands creating empty layers that get optimized out in the cache. There is a known issue that in some cases this might cause history go out of sync, but I'm not aware of any case where the actual layer would get lost. I don't really have a solution for that problem and I think we need to remove the empty layers removal optimization.

tonistiigi commented 3 years ago

There are no special "cache debug" options atm. without code modification.

mbentley commented 3 years ago

Interesting. Well thanks for the insights into a possibly why it could be happening. I hadn't tried to look further into a previous build's output where it would have pushed the cache and inspected the specific image cache that is getting pushed to my registry for anything that sticks out.

Unfortunately I am not doing so well at reproducing the issue. I've tried all sort of various combinations of existing builder, existing builder with builder cache cleared and creating builders from scratch and looping through on builds. I think I've probably built this image about 1000 times 😄 There doesn't seem to be any real reason or trigger as to why it happens as far as I can tell.

If you'd rather I close this, that's fine. I'll just keep looking to see if there is anything I can do to try to figure out what is going on and how I can trigger it.

mbentley commented 3 years ago

@tonistiigi - any chance you know approximately when that sort of behavior might have been introduced? I was chatting with a former coworker and he had been using buildkit 0.7.1 so I tried that out and I have currently built 80 images via my CI five times over without the issue coming up which is more than when I was running 0.8.3 it felt like but I just added a script to try to detect these sort of issues today after each build. I see a number of mentions of cache enhancements throughout the history; maybe most significantly in 0.8.0 and above but I was hoping that I might be able to narrow it down to some specific version(s) without having to step through each release to figure out where the issue might have been introduced.

Patrick-Remy commented 3 years ago

Sounds really like the problem I described in https://github.com/moby/buildkit/issues/1980#issuecomment-786280820 that commit https://github.com/moby/buildkit/commit/dda009a58c76e3e54b4539dce23623eefcc4428c broke it, also needed to downgrade to 0.7

I would really appreciate if the behavior is changed in the next Bugfix release, as it is critical if those images were tried to deployed on prod. We wrote test scripts to prevent this.

tonistiigi commented 3 years ago

@Patrick-Remy We need a reproducer before anything can be concluded from this. If you have one please post it.

Patrick-Remy commented 3 years ago

I did already in my issue: https://github.com/Patrick-Remy/buildkit-missing-layer-repro

tonistiigi commented 3 years ago

@Patrick-Remy Tried to run in both with v0.8.1 and v0.8.3 but doesn't look like it is doing anything weird for me. Logs https://gist.github.com/tonistiigi/f135e38d25d0d204985d9659f19888ed

Patrick-Remy commented 3 years ago

I just could simplify the setup massively, based on @mbentley's provided repo. I updated my repo https://github.com/Patrick-Remy/buildkit-missing-layer-repro. The build-script.sh now finishes in a minute and I could reproduce it on several machines. Please give it a second try.

mbentley commented 3 years ago

Nice, thanks @Patrick-Remy! I've been able to use your repro myself on Docker for Mac and it takes anywhere from 10-30 seconds fairly reliably to get it to reproduce the issue.

Just to add a quick note, I did test a number of versions with your script to reproduce it (v0.8.3 v0.8.2 v0.8.1 v0.8.0 v0.7.2 v0.7.1) and found that the issue does first start appearing as of 0.8.0.

tonistiigi commented 3 years ago

I can confirm that I seem to be able to repro with the latest version.

Patrick-Remy commented 3 years ago

Just to add a quick note, I did test a number of versions with your script to reproduce it (v0.8.3 v0.8.2 v0.8.1 v0.8.0 v0.7.2 v0.7.1) and found that the issue does first start appearing as of 0.8.0.

Yes, as I wrote in the README I already debugged it to be caused by this commit (I built versions myself for commits between 0.8.0 and 0.7.2): https://github.com/moby/buildkit/commit/dda009a58c76e3e54b4539dce23623eefcc4428c

I can confirm that I seem to be able to repro with the latest version.

That's great! I look forward that you will find the possible bug in the implementation ;)

tonistiigi commented 3 years ago

I made one PR regarding the repro. https://github.com/moby/buildkit/pull/2261 It fixes the repro but obviously is not the main issue here. This has left me quite puzzled how changing trivial things in the repro changes the behavior. The 1 and 2 caches contain identical files but it does not work if 2 does not run. Even if I clear the local state and now 2 should have no effect it still changes behavior. I'll continue to look into this but we have hit the deadline with the release and need to move on there.

Patrick-Remy commented 2 years ago

Did you have any time to investigate this another time? We are still freezed to v0.7.2 due to this bug.

jgiannuzzi commented 2 years ago

I have investigated this issue further and made another repro, based on @Patrick-Remy's excellent work. It does not need to import 2 identical caches to make the issue appear, and has an even shorter Dockerfile.

I summarised my understanding of the issue in the repro README (mind you, I have never looked at the buildkit codebase before, so this might be wrong) and copied it below for convenience.

Why is this broken?

Let's take a look at the Dockerfile provided in this repository:

FROM alpine:latest

# create a layer (empty or not)
RUN echo 1

# create a layer that also depends on the context
COPY repro.txt /

# create an empty layer
RUN echo 2

When importing the cache of a run that has empty layers removed, some vertexes will point to the same result, e.g. COPY repro.txt / and RUN echo 2.

In cache.remotecache.v1.(*cacheResultStorage).LoadWithParents, we try to load a cache result with its parents. We start by looking up the corresponding item in a map, and because there are 2 possible values, it will randomly return one or the other.

If the 'wrong' item gets used (COPY repro.txt / in our example), then only a partial list of results will be loaded. They get returned to solver.(*cacheManager).LoadWithParents, which will filter them and end up with the same partial list of results.

Those will eventually be saved in the buildkitd cache in solver.(*combinedCacheManager).Load, thus missing the entry for RUN echo 2.

During a second run with the same cache, but this time with a partially populated buildkitd cache, if the 'wrong' item gets used again in cache.remotecache.v1.(*cacheResultStorage).LoadWithParents, and the partial list of results is loaded and returned to solver.(*cacheManager).LoadWithParents, something different from the previous run might happen.

During the result filtering, results originating from both caches could be walked, and the result for RUN echo 1 could end up being returned as the first element of the list, instead of the one for COPY repro.txt / or RUN echo 2.

Unfortunately, solver.(*combinedCacheManager).Load assumes that the first result is the parent and will return that one, which eventually results in an image missing a layer!

mbentley commented 2 years ago

I've had the chance to do a couple re-builds of all 90 of my images that I push to Docker Hub using buildkit from your PR @jgiannuzzi and while it is a small sample size of two full runs, I am not seeing any zero bytes layers being detected 🤞