wolf-base breakage due to delta on docker hub vs chain guard registry

[kapilt]

Which image/versions are related to this issue/feature request?

wolfi-base:latest

Issue/Feature description

Given a simple docker file below, using wolfi-base from docker hub appears to be broken, while using it from chain guard registry works...

# XXXX flip these to reproduce locally
FROM chainguard/wolfi-base as builder
FROM cgr.dev/chainguard/wolfi-base as builder

ARG POETRY_VERSION="1.8.3"
ARG PY_VERSION=3.12
WORKDIR /app

RUN apk add python-${PY_VERSION} py${PY_VERSION}-pip && \
    chown -R nonroot.nonroot /app/

USER nonroot

# Install Poetry in its own virtual environment
RUN python3 -m venv "${HOME}/tools" && \
    . "${HOME}/tools/bin/activate" && \
    pip install --verbose "poetry==${POETRY_VERSION}"

building against chain guard registry, no issues, but building against docker hub registry

❯ docker build --no-cache -t c7n-left -f docker/c7n-left .
[+] Building 15.9s (9/19)                                                                                                                                                                                                                                                                             docker:orbstack
 => [internal] load build definition from c7n-left                                                                                                                                                                                                                                                               0.0s
 => => transferring dockerfile: 1.87kB                                                                                                                                                                                                                                                                           0.0s
 => [internal] load metadata for docker.io/chainguard/wolfi-base:latest                                                                                                                                                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                0.0s
 => => transferring context: 376B                                                                                                                                                                                                                                                                                0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                0.0s
 => => transferring context: 12.61kB                                                                                                                                                                                                                                                                             0.0s
 => [stage-1 1/7] FROM docker.io/chainguard/wolfi-base:latest                                                                                                                                                                                                                                                    0.0s
 => CACHED [stage-1 2/7] WORKDIR /app                                                                                                                                                                                                                                                                            0.0s
 => [builder  3/10] RUN apk add python-3.12 py3.12-pip &&     chown -R nonroot.nonroot /app/                                                                                                                                                                                                                     5.7s
 => [stage-1 3/7] RUN apk add git python-3.12 &&     chown -R nonroot.nonroot /app/                                                                                                                                                                                                                              6.6s
 => ERROR [builder  4/10] RUN python3 -m venv "${HOME}/tools" &&     . "${HOME}/tools/bin/activate" &&     pip install --verbose "poetry==1.8.3"                                                                                                                                                                10.1s
------                                                                                                                                                                                                                                                                                                                
 > [builder  4/10] RUN python3 -m venv "${HOME}/tools" &&     . "${HOME}/tools/bin/activate" &&     pip install --verbose "poetry==1.8.3":                                                                                                                                                                            
2.345 Using pip 24.0 from /home/nonroot/tools/lib/python3.12/site-packages/pip (python 3.12)                                                                                                                                                                                                                          
2.365 WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.                                                                                                                                                                                              
2.368 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/poetry/                                                                                           
2.872 WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/poetry/                                                                                           
3.876 WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/poetry/
5.879 WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/poetry/
9.881 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/poetry/
9.883 Could not fetch URL https://pypi.org/simple/poetry/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/poetry/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
9.893 ERROR: Could not find a version that satisfies the requirement poetry==1.8.3 (from versions: none)                                                   
9.893 ERROR: No matching distribution found for poetry==1.8.3
9.904 WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
9.908 Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
------
c7n-left:14
--------------------
  13 |     # Install Poetry in its own virtual environment
  14 | >>> RUN python3 -m venv "${HOME}/tools" && \
  15 | >>>     . "${HOME}/tools/bin/activate" && \
  16 | >>>     pip install --verbose "poetry==${POETRY_VERSION}"
  17 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c python3 -m venv \"${HOME}/tools\" &&     . \"${HOME}/tools/bin/activate\" &&     pip install --verbose \"poetry==${POETRY_VERSION}\"" did not complete successfully: exit code: 1

what gives?

[found-it]

Hey @kapilt, thanks for bringing this to our attention. I tried to reproduce locally but was able to get both images to build (attached my terminal log). There could be a difference in our build environments though, so I'd like to check a few things.

Click to expand terminal log

```sh /private/tmp/wolfi-repro via 🐳 desktop-linux took 11s ➜ cat Dockerfile # XXXX flip these to reproduce locally FROM chainguard/wolfi-base as builder # FROM cgr.dev/chainguard/wolfi-base as builder ARG POETRY_VERSION="1.8.3" ARG PY_VERSION=3.12 WORKDIR /app RUN apk add python-${PY_VERSION} py${PY_VERSION}-pip && \ chown -R nonroot.nonroot /app/ USER nonroot # Install Poetry in its own virtual environment RUN python3 -m venv "${HOME}/tools" && \ . "${HOME}/tools/bin/activate" && \ pip install --verbose "poetry==${POETRY_VERSION}" /private/tmp/wolfi-repro via 🐳 desktop-linux took 2s ➜ docker build --no-cache -t localhost/repro:dockerhub . [+] Building 11.6s (9/9) FINISHED docker:desktop-linux => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 517B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/chainguard/wolfi-base:latest 0.8s => [auth] chainguard/wolfi-base:pull token for registry-1.docker.io 0.0s => [1/4] FROM docker.io/chainguard/wolfi-base@sha256:3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92 0.0s => CACHED [2/4] WORKDIR /app 0.0s => [3/4] RUN apk add python-3.12 py3.12-pip && chown -R nonroot.nonroot /app/ 2.0s => [4/4] RUN python3 -m venv "${HOME}/tools" && . "${HOME}/tools/bin/activate" && pip install --verbose "poetry==1.8.3" 8.5s => exporting to image 0.2s => => exporting layers 0.2s => => writing image sha256:54d35bd5aec51af4f1d9f8e660c2772df20e071d4e21b9b47021ff73f1fc1b10 0.0s => => naming to localhost/repro:dockerhub 0.0s /private/tmp/wolfi-repro via 🐳 desktop-linux ➜ cat Dockerfile # XXXX flip these to reproduce locally # FROM chainguard/wolfi-base as builder FROM cgr.dev/chainguard/wolfi-base as builder ARG POETRY_VERSION="1.8.3" ARG PY_VERSION=3.12 WORKDIR /app RUN apk add python-${PY_VERSION} py${PY_VERSION}-pip && \ chown -R nonroot.nonroot /app/ USER nonroot # Install Poetry in its own virtual environment RUN python3 -m venv "${HOME}/tools" && \ . "${HOME}/tools/bin/activate" && \ pip install --verbose "poetry==${POETRY_VERSION}" /private/tmp/wolfi-repro via 🐳 desktop-linux ➜ docker build --no-cache -t localhost/repro:cgr . [+] Building 11.0s (9/9) FINISHED docker:desktop-linux => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 517B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for cgr.dev/chainguard/wolfi-base:latest 0.5s => [auth] chainguard/wolfi-base:pull token for cgr.dev 0.0s => [1/4] FROM cgr.dev/chainguard/wolfi-base@sha256:3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92 0.0s => CACHED [2/4] WORKDIR /app 0.0s => [3/4] RUN apk add python-3.12 py3.12-pip && chown -R nonroot.nonroot /app/ 1.8s => [4/4] RUN python3 -m venv "${HOME}/tools" && . "${HOME}/tools/bin/activate" && pip install --verbose "poetry==1.8.3" 8.5s => exporting to image 0.2s => => exporting layers 0.2s => => writing image sha256:0ee8c11e9858fdf8eecbdfe330d103d8f08a039c75f17d4eb18a453efbfe337a 0.0s => => naming to localhost/repro:cgr 0.0s /private/tmp/wolfi-repro via 🐳 desktop-linux took 11s ➜ ```

Could you confirm the digest for the docker hub image matches the digest for the cgr image? The crane command makes it easy

/private/tmp/wolfi-repro via 🐳 desktop-linux took 11s
➜ crane digest cgr.dev/chainguard/wolfi-base
sha256:3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92

/private/tmp/wolfi-repro via 🐳 desktop-linux
➜ crane digest chainguard/wolfi-base
sha256:3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92

[kapilt]

confirmed on image hash matches (output below).. so a bit baffled. on an arm64 re mac, I started going down this road due to our only wolfi based image which uses public docker hub registry (vs commercial chain guard registry) started timing out on cross compile from GitHub actions over the last few weeks on our one image that was using wolfi-base vs ubuntu and was trying to debug locally when I and ran into this.

❯ crane digest cgr.dev/chainguard/wolfi-base
sha256:3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92

❯ crane digest chainguard/wolfi-base
sha256:3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92

[kapilt]

also re wolfi based image having weird timeouts, this is an oss project, so here's the timeline on the build starting to fail, its all timeout based. either 14m for cross arch docker image build to be good or fail on the one wolfi image (c7n-left) after 6 hrs, at a slightly different step of doing poetry install.
, https://github.com/cloud-custodian/cloud-custodian/actions/workflows/docker.yml

anyways.. I can still reproduce the originally reported error if you want any more info.

[jdolitsky]

In the logs from latest cloud-custodian build, I noticed the use of docker buildx build:

/usr/bin/docker buildx build --build-arg POETRY_VERSION=1.5.1 
...

It seems somewhat likely to be related 🤷

[found-it]

Interesting, good to know. I cloned the cloud-custodian project and still built the c7n-left image successfully with

docker build --no-cache -t c7n-left -f docker/c7n-left .

I don't see a reference to the image sha in your local build command, all I see is

...
 => [stage-1 1/7] FROM docker.io/chainguard/wolfi-base:latest
...

Can you double check you are building with the latest wolfi-base image locally? I can see CI uses the 3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92 sha but as you said, it's timing out rather than throwing an SSL error like you see locally

➜ docker images --digests | grep chainguard/wolfi-base
chainguard/wolfi-base                         latest                                     sha256:3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92   91668ee15d7f   4 days ago          13.4MB

➜ docker pull chainguard/wolfi-base
Using default tag: latest
latest: Pulling from chainguard/wolfi-base
Digest: sha256:3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92
Status: Image is up to date for chainguard/wolfi-base:latest
docker.io/chainguard/wolfi-base:latest

[kapilt]

doh, that was it stale local image, aka user error :/ sorry about the noise. closing this.

the ci issue is indeed separate, its a bit unclear to me why it only exhibits on our wolfi based image, but I'll get back to debugging that.

❯ docker images --digests | grep chainguard/wolfi-base
cgr.dev/chainguard/wolfi-base            latest                                                             sha256:3eff851ab805966c768d2a8107545a96218426cee1e5cc805865505edbe6ce92   91668ee15d7f   4 days ago      13.4MB
chainguard/wolfi-base                    latest                                                             sha256:07d99e3cca939979cbfaa458b702a8910e55f0b6e6a68a2a8ec5ae41f2d9e639   b91dd14cdbfa   2 months ago    31.2MB