WARNING: opening from cache https://mirrors.ustc.edu.cn/alpine/v3.19/main: UNTRUSTED signature

zw963 commented 9 months ago

问题描述 / Bug description

不确定是我本地网络的问题还是 mirrors 的问题，三四天前还可以用的。

请参考如下 log:

#10 [base 3/6] RUN --mount=type=cache,target=/var/cache/apk     set -eux;     apk add     --update     ca-certificates     yaml-dev yaml-static     openssl-dev openssl-libs-static     zlib-dev zlib-static     libunwind-dev libunwind-static     libssh2-dev libssh2-static     lz4-dev lz4-static     sqlite-static     libxml2-dev     tzdata     crystal shards     ;
#10 0.025 + apk add --update ca-certificates yaml-dev yaml-static openssl-dev openssl-libs-static zlib-dev zlib-static libunwind-dev libunwind-static libssh2-dev libssh2-static lz4-dev lz4-static sqlite-static libxml2-dev tzdata crystal shards
#10 0.028 fetch https://mirrors.ustc.edu.cn/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
#10 1.250 WARNING: opening from cache https://mirrors.ustc.edu.cn/alpine/v3.19/main: UNTRUSTED signature
#10 1.250 fetch https://mirrors.ustc.edu.cn/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
#10 2.210 WARNING: opening from cache https://mirrors.ustc.edu.cn/alpine/v3.19/community: UNTRUSTED signature
#10 2.211 ERROR: unable to select packages:
#10 2.211   ca-certificates (no such package):
#10 2.211     required by: world[ca-certificates]
#10 2.211   crystal (no such package):
#10 2.211     required by: world[crystal]
#10 2.211   libssh2-dev (no such package):
#10 2.211     required by: world[libssh2-dev]
#10 2.211   libssh2-static (no such package):
#10 2.211     required by: world[libssh2-static]
#10 2.211   libunwind-dev (no such package):
#10 2.211     required by: world[libunwind-dev]
#10 2.211   libunwind-static (no such package):
#10 2.211     required by: world[libunwind-static]
#10 2.211   libxml2-dev (no such package):
#10 2.211     required by: world[libxml2-dev]
#10 2.211   lz4-dev (no such package):
#10 2.211     required by: world[lz4-dev]
#10 2.211   lz4-static (no such package):
#10 2.211     required by: world[lz4-static]
#10 2.211   openssl-dev (no such package):
#10 2.211     required by: world[openssl-dev]
#10 2.211   openssl-libs-static (no such package):
#10 2.211     required by: world[openssl-libs-static]
#10 2.211   shards (no such package):
#10 2.211     required by: world[shards]
#10 2.211   sqlite-static (no such package):
#10 2.211     required by: world[sqlite-static]
#10 2.211   tzdata (no such package):
#10 2.211     required by: world[tzdata]
#10 2.211   yaml-dev (no such package):
#10 2.211     required by: world[yaml-dev]
#10 2.211   yaml-static (no such package):
#10 2.211     required by: world[yaml-static]
#10 2.211   zlib-dev (no such package):
#10 2.211     required by: world[zlib-dev]
#10 2.211   zlib-static (no such package):
#10 2.211     required by: world[zlib-static]
#10 ERROR: process "/bin/sh -c set -eux;     apk add     --update     ca-certificates     yaml-dev yaml-static     openssl-dev openssl-libs-static     zlib-dev zlib-static     libunwind-dev libunwind-static     libssh2-dev libssh2-static     lz4-dev lz4-static     sqlite-static     libxml2-dev     tzdata     crystal shards     ;" did not complete successfully: exit code: 18

#11 [link_target 3/7] RUN --mount=type=cache,target=/var/cache/apk     set -ux;     apk add     --update     gc-dev pcre2-dev libevent-static     gcc make automake libtool autoconf curl git     ;
#11 0.037 + apk add --update gc-dev pcre2-dev libevent-static gcc make automake libtool autoconf curl git
#11 0.068 fetch https://mirrors.ustc.edu.cn/alpine/v3.19/main/aarch64/APKINDEX.tar.gz
#11 1.473 WARNING: updating https://mirrors.ustc.edu.cn/alpine/v3.19/main: No such file or directory
#11 1.473 fetch https://mirrors.ustc.edu.cn/alpine/v3.19/community/aarch64/APKINDEX.tar.gz
#11 CANCELED

我使用的 build 参数如下：

docker buildx build --progress=plain --no-cache ...

谢谢。

iBug commented 9 months ago

能否提供 Dockerfile 方便我们复现？

taoky commented 9 months ago

无法复现：

> docker run -it --rm --platform=linux/arm64 alpine:3.19
Unable to find image 'alpine:3.19' locally
3.19: Pulling from library/alpine
c30352492317: Pull complete 
Digest: sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
Status: Downloaded newer image for alpine:3.19
/ # sed -i "s/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g" /etc/apk/repositories
/ # apk update
fetch https://mirrors.ustc.edu.cn/alpine/v3.19/main/aarch64/APKINDEX.tar.gz
fetch https://mirrors.ustc.edu.cn/alpine/v3.19/community/aarch64/APKINDEX.tar.gz
v3.19.0-240-g97aef46a80c [https://mirrors.ustc.edu.cn/alpine/v3.19/main]
v3.19.0-239-gfd557282b4a [https://mirrors.ustc.edu.cn/alpine/v3.19/community]
OK: 22843 distinct packages available
/ # apk add ca-certificates
(1/1) Installing ca-certificates (20230506-r0)
Executing busybox-1.36.1-r15.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 8 MiB in 16 packages
/ #

zw963 commented 9 months ago

能否提供 Dockerfile 方便我们复现？

我的 Dockerfile 文件

```dockerfile # -*- mode: dockerfile-ts; -*- # 确保 build 和 linking 是同一个 alpine 版本 ARG alpine_version=3.19 # mirrors.ustc.edu.cn mirrors.tuna.tsinghua.edu.cn ARG alpine_mirror=mirrors.ustc.edu.cn # =============== 使用本地主机相同的架构 cross compile =============== # 如果这里不指定 --platform, 则 docker 会自动选择根据命令行 --platform 指定的架构 # 这里强制使用当前 build 所在平台的架构，来高性能的 cross compile. FROM --platform=$BUILDPLATFORM alpine:$alpine_version AS base ARG alpine_mirror RUN sed -i "s/dl-cdn.alpinelinux.org/$alpine_mirror/g" /etc/apk/repositories # Add trusted CAs for communicating with external services and required build tooling # 额外添加了 -dev 包，也允许编译动态版本。 # crystal 依赖：libevent-dev libevent-static llvm16-libs musl-dev pcre2-dev gc-dev gcc RUN --mount=type=cache,target=/var/cache/apk \ set -eux; \ apk add \ --update \ ca-certificates \ yaml-dev yaml-static \ openssl-dev openssl-libs-static \ zlib-dev zlib-static \ libunwind-dev libunwind-static \ libssh2-dev libssh2-static \ lz4-dev lz4-static \ sqlite-static \ libxml2-dev \ tzdata \ crystal shards \ ; # 为了 https 的情况下，将证书一起部署可以工作，这个似乎是必须的。 RUN update-ca-certificates # RUN --mount=type=cache,target=/var/cache/apk \ # set -eux; \ # apk add \ # --update \ # --repository=http://$alpine_mirror/alpine/edge/main \ # --repository=http://$alpine_mirror/alpine/edge/community \ # crystal \ # shards RUN addgroup -g 1000 docker && \ adduser -u 1000 -G docker -h /home/docker -s /bin/sh -D docker USER docker:docker # WORKDIR 应该放在 USER 后面，因为如果 /app 不存在，需要使用 user docker 创建这个文件夹 WORKDIR /app FROM base AS build_cross_platform ARG TARGETARCH # Install shards for caching COPY shard.yml shard.yml COPY shard.lock shard.lock RUN shards install --production --ignore-crystal-version \ --skip-postinstall --skip-executables COPY src src RUN mkdir -p bin && \ shards build --production --release --no-debug \ --error-trace --static --stats --time \ -Dstrict_multi_assign -Dno_number_autocast -Duse_pcre2 -Dno_restrictions_augmenter \ --cross-compile --target=$TARGETARCH |tail -1 > bin/app.sh SHELL ["/bin/ash", "-eo", "pipefail", "-c"] # Extract binary dependencies (uncomment if not compiling a static build) RUN for binary in /app/bin/*; do \ if ldd=$(ldd $binary 2>/dev/null); then \ echo "$ldd" |tr -s '[:blank:]' '\n' |grep '^/' |xargs -I % sh -c 'mkdir -p $(dirname deps%); cp % deps%;'; \ mv $binary ${binary}_real; \ mv deps bin; \ echo "#!/usr/bin/env sh" > $binary; \ echo 'ROOT=${0%/*}' >> $binary; \ echo 'LD_LIBRARY_PATH=$ROOT/deps/lib:$ROOT/deps/usr/lib/ $ROOT/app_real' >> $binary; \ chmod +x $binary; \ else \ true; \ fi \ done # ============================== 使用目标系统架构 link binary ============================== FROM alpine:$alpine_version as link_target ARG alpine_mirror RUN sed -i "s/dl-cdn.alpinelinux.org/$alpine_mirror/g" /etc/apk/repositories # Add trusted CAs for communicating with external services and required build tooling # 额外添加了 -dev 包，也允许编译动态版本。 # g++ 依赖 libc-dev, libc-dev 依赖 musl-dev RUN --mount=type=cache,target=/var/cache/apk \ set -ux; \ apk add \ --update \ gc-dev pcre2-dev libevent-static \ gcc make automake libtool autoconf curl git \ ; # 为了 https 的情况下，将证书一起部署可以工作，这个似乎是必须的。 # RUN update-ca-certificates RUN addgroup -g 1000 docker && \ adduser -u 1000 -G docker -h /home/docker -s /bin/sh -D docker USER docker:docker # WORKDIR 应该放在 USER 后面，因为如果 /app 不存在，需要创建这个文件夹 WORKDIR /app # All new files and directories are created with a UID and GID of 0, unless the optional --chown flag COPY --from=build_cross_platform --chown=docker:docker /app/bin bin RUN sh -ex bin/app.sh # Build a minimal docker image FROM scratch AS mini WORKDIR / ENV PATH=$PATH:/ # Copy the user information over COPY --from=build_cross_platform /etc/passwd /etc/passwd COPY --from=build_cross_platform /etc/group /etc/group # These are required for communicating with external services COPY --from=build_cross_platform /etc/hosts /etc/hosts # These provide certificate chain validation where communicating with external services over TLS COPY --from=build_cross_platform /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt # This is required for Timezone support COPY --from=build_cross_platform /usr/share/zoneinfo/ /usr/share/zoneinfo/ # This is your application COPY --from=link_target /app/bin / # COPY --from=build_cross_platform /app/openapi.yml /openapi.yml USER docker:docker # Spider-gazelle has a built in helper for health checks (change this as desired for your applications) # HEALTHCHECK CMD ["/app", "-c", "http://127.0.0.1:3000/"] # Run the app binding on port 3000 # EXPOSE 3000 ENTRYPOINT ["/app/bin/app"] # CMD ["/app", "-b", "0.0.0.0", "-p", "3000"] ```

最后一个 staging 可能还没很好工作，但是不用管它。

我是这样运行的。

target="aarch64-linux-musl" arch="linux/arm64/v8"

docker buildx build --progress=plain --no-cache --build-arg TARGETARCH=$target --platform $arch --target mini -t crystal_build_staticbinary$arch --output type=local,dest=$arch -f "上面的文件" .

谢谢。

iBug commented 9 months ago

根据你的 log 输出，我复制了你的 Dockerfile 前面 4 个步骤，然后 docker build -t t .，其中 [base 3/4] 步骤正常运行，无法复现你的问题。

我的 log

```text #0 building with "default" instance using docker driver #1 [internal] load .dockerignore #1 transferring context: 2B done #1 DONE 0.0s #2 [internal] load build definition from Dockerfile #2 transferring dockerfile: 1.41kB done #2 DONE 0.0s #3 [internal] load metadata for docker.io/library/alpine:3.19 #3 DONE 1.0s #4 [base 1/4] FROM docker.io/library/alpine:3.19@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48 #4 CACHED #5 [base 2/4] RUN sed -i "s/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g" /etc/apk/repositories #5 DONE 0.2s #6 [base 3/4] RUN --mount=type=cache,target=/var/cache/apk set -eux; apk add --update ca-certificates yaml-dev yaml-static openssl-dev openssl-libs-static zlib-dev zlib-static libunwind-dev libunwind-static libssh2-dev libssh2-static lz4-dev lz4-static sqlite-static libxml2-dev tzdata crystal shards ; #6 0.392 + apk add --update ca-certificates yaml-dev yaml-static openssl-dev openssl-libs-static zlib-dev zlib-static libunwind-dev libunwind-static libssh2-dev libssh2-static lz4-dev lz4-static sqlite-static libxml2-dev tzdata crystal shards #6 0.398 fetch https://mirrors.ustc.edu.cn/alpine/v3.19/main/x86_64/APKINDEX.tar.gz #6 0.476 fetch https://mirrors.ustc.edu.cn/alpine/v3.19/community/x86_64/APKINDEX.tar.gz #6 0.644 (1/87) Upgrading libcrypto3 (3.1.4-r2 -> 3.1.4-r3) #6 0.662 (2/87) Upgrading libssl3 (3.1.4-r2 -> 3.1.4-r3) #6 0.666 (3/87) Installing ca-certificates (20230506-r0) ... #6 2.414 (85/87) Installing yaml-dev (0.2.5-r2) #6 2.415 (86/87) Installing yaml-static (0.2.5-r2) #6 2.416 (87/87) Installing zlib-static (1.3-r2) #6 2.419 Executing busybox-1.36.1-r15.trigger #6 2.422 Executing ca-certificates-20230506-r0.trigger #6 2.461 OK: 424 MiB in 100 packages #6 DONE 2.9s #7 [base 4/4] RUN update-ca-certificates #7 DONE 0.4s #8 exporting to image #8 exporting layers #8 exporting layers 2.0s done #8 writing image sha256:4dd603d6520a68288728033e562da9e85528da45e1e0261658487cf56d9603d1 done #8 naming to docker.io/library/t done #8 DONE 2.0s ```

是否可以重新提供一个最小可复现样例？你的 Dockerfile 过于庞大，而且你的 buildx 命令我无法直接使用：

$ docker buildx build --progress=plain --no-cache --build-arg TARGETARCH=$target --platform $arch --target mini -t crystal_build_static_binary_$arch --output type=local,dest=$arch .
#0 building with "default" instance using docker driver

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.41kB done
#2 DONE 0.0s
ERROR: failed to solve: target stage mini could not be found

taoky commented 9 months ago

我怀疑是因为构建 base 和 link_target 是同时进行的，并且都挂载了 cache mount，导致有两个 apk 同时在写数据（导致损坏），是该 Dockerfile 写法上出现的并发问题，与镜像站无关。

zw963 commented 9 months ago

多谢，我刚想起来，最近更改过启动 dockerd 的 --registry-mirror=??? 参数，我换了一个新的 mirror, docker_mirror=https://mirror.ccs.tencentyun.com, 本地重新启动了下 dockerd, 似乎可以了。

zw963 commented 9 months ago

不确定是不是官方源问题，哈，所以在这里提一下。

#10 [link_target 3/8] RUN --mount=type=cache,target=/var/cache/apk     set -ux;     apk add     --update     gc-dev pcre2-dev libevent-static     sqlite-static openssl-dev openssl-libs-static     zlib-dev zlib-static     g++ make automake libtool autoconf curl git     ;
#10 0.034 + apk add --update gc-dev pcre2-dev libevent-static sqlite-static openssl-dev openssl-libs-static zlib-dev zlib-static g++ make automake libtool autoconf curl git
#10 0.069 fetch https://mirrors.ustc.edu.cn/alpine/v3.19/main/aarch64/APKINDEX.tar.gz
#10 1.089 WARNING: updating https://mirrors.ustc.edu.cn/alpine/v3.19/main: No such file or directory
#10 1.089 fetch https://mirrors.ustc.edu.cn/alpine/v3.19/community/aarch64/APKINDEX.tar.gz
#10 CANCELED

#11 [base 3/6] RUN --mount=type=cache,target=/var/cache/apk     set -eux;     apk add     --update     ca-certificates     yaml-dev yaml-static     openssl-dev openssl-libs-static     zlib-dev zlib-static     libunwind-dev libunwind-static     libssh2-dev libssh2-static     lz4-dev lz4-static     sqlite-static     libxml2-dev     tzdata     crystal shards     ;
#11 0.020 + apk add --update ca-certificates yaml-dev yaml-static openssl-dev openssl-libs-static zlib-dev zlib-static libunwind-dev libunwind-static libssh2-dev libssh2-static lz4-dev lz4-static sqlite-static libxml2-dev tzdata crystal shards
#11 0.023 fetch https://mirrors.ustc.edu.cn/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
#11 0.812 fetch https://mirrors.ustc.edu.cn/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
#11 1.243 WARNING: opening from cache https://mirrors.ustc.edu.cn/alpine/v3.19/community: IO ERROR
#11 1.256 ERROR: unable to select packages:
#11 1.256   crystal (no such package):
#11 1.256     required by: world[crystal]
#11 1.256   shards (no such package):
#11 1.256     required by: world[shards]

正如你看到的，有两行有错误。

#10 1.089 WARNING: updating https://mirrors.ustc.edu.cn/alpine/v3.19/main: No such file or directory
...
#11 1.243 WARNING: opening from cache https://mirrors.ustc.edu.cn/alpine/v3.19/community: IO ERROR

并且都挂载了 cache mount，导致有两个 apk 同时在写数据（导致损坏），

如果这真的是这个问题，但是我将 mirrors.ustc.edu.cn 替换回 dl-cdn.alpinelinux.org 就立刻工作的很好了，奇怪。

zw963 commented 9 months ago

抱歉, 可能还是本地网络问题，又跑了一次，问题再次出现。

taoky commented 9 months ago

不确定是不是官方源问题，哈，所以在这里提一下。

请参考 https://github.com/ustclug/discussions/issues/442#issuecomment-1891768164，如果无法复现的话，那么基本可以说明不是镜像站的问题。

如果这真的是这个问题，但是我将 mirrors.ustc.edu.cn 替换回 dl-cdn.alpinelinux.org 就立刻工作的很好了，奇怪。

并发问题是存在随机性的，与此同时与很多因素都有关。一种可能的解释是，dl-cdn.alpinelinux.org 下载更慢，所以更不容易触发问题。

我相信在上面的回复中我已经明确指出了你的 Dockerfile 问题所在（并且修复的方式也很明显——不要自己搞 cache，或者想办法让它们不要同时进行构建），之后除非有更新的、能够证明不是并发问题的信息，否则恐怕我没有再回复的必要了。

zw963 commented 9 months ago

不确定是不是官方源问题，哈，所以在这里提一下。

请参考 #442 (comment)，如果无法复现的话，那么基本可以说明不是镜像站的问题。

如果这真的是这个问题，但是我将 mirrors.ustc.edu.cn 替换回 dl-cdn.alpinelinux.org 就立刻工作的很好了，奇怪。

并发问题是存在随机性的，与此同时与很多因素都有关。一种可能的解释是，dl-cdn.alpinelinux.org 下载更慢，所以更不容易触发问题。

我相信在上面的回复中我已经明确指出了你的 Dockerfile 问题所在（并且修复的方式也很明显——不要自己搞 cache，或者想办法让它们不要同时进行构建），之后除非有更新的、能够证明不是并发问题的信息，否则恐怕我没有再回复的必要了。

无论如何，多谢！经过很多次尝试，基本确定，就是 cache mount 造成的，现在我全部移除了，用了很多次，但没出过问题。

RUN --mount=type=cache,target=/var/cache/apk，完全是抄来的，不懂瞎用，自己给自己挖坑，汗～其实，build staging 根本没必要 cache, 反正会扔掉。

ustclug / discussions

WARNING: opening from cache https://mirrors.ustc.edu.cn/alpine/v3.19/main: UNTRUSTED signature #442

问题描述 / Bug description