ADD https://ssl-config.mozilla.org/ffdhe2048.txt /etc/nginx/ssl-dhparams.pem: ------ failed to solve: failed to load cache key: invalid not-modified ETag: "65a0156d-1a7"

[macdjord]

My build, which was working fine previously, is failing because of an error with a cache key:

macdjord@MYWORKMACHINE:~/source/OURPROJECT.git$ docker compose build nginx-frontend
WARN[0000] The "CI_JOB_ID" variable is not set. Defaulting to a blank string.
WARN[0000] The "CI_PIPELINE_ID" variable is not set. Defaulting to a blank string.
WARN[0000] The "CI_JOB_ID" variable is not set. Defaulting to a blank string.
WARN[0000] The "CI_PIPELINE_ID" variable is not set. Defaulting to a blank string.
WARN[0000] The "CI_PROJECT_NAME" variable is not set. Defaulting to a blank string.
WARN[0000] The "CI_COMMIT_BRANCH" variable is not set. Defaulting to a blank string.
WARN[0000] The "CI_COMMIT_SHA" variable is not set. Defaulting to a blank string.
WARN[0000] The "GIT_COMMIT_TIMESTAMP_UTC" variable is not set. Defaulting to a blank string.
#0 building with "default" instance using docker driver

#1 [nginx-frontend internal] load build definition from Dockerfile
#1 transferring dockerfile: 516B 0.0s done
#1 DONE 0.1s

#2 [nginx-frontend internal] load metadata for docker.io/library/nginx@sha256:a45ee5d042aaa9e81e013f97ae40c3dda26fbe98f22b6251acdf28e579560d55
#2 DONE 0.0s

#3 [nginx-frontend internal] load .dockerignore
#3 transferring context: 2B 0.0s done
#3 DONE 0.1s

#4 [nginx-frontend internal] load build context
#4 DONE 0.0s

#5 [nginx-frontend 1/4] FROM docker.io/library/nginx@sha256:a45ee5d042aaa9e81e013f97ae40c3dda26fbe98f22b6251acdf28e579560d55
#5 CACHED

#6 [nginx-frontend 2/4] ADD https://ssl-config.mozilla.org/ffdhe2048.txt /etc/nginx/ssl-dhparams.pem
#6 ...

#4 [nginx-frontend internal] load build context
#4 transferring context: 137B 0.1s done
#4 DONE 0.1s

#6 [nginx-frontend 2/4] ADD https://ssl-config.mozilla.org/ffdhe2048.txt /etc/nginx/ssl-dhparams.pem
#6 ERROR: invalid not-modified ETag: "65a0156d-1a7"
------
 > [nginx-frontend 2/4] ADD https://ssl-config.mozilla.org/ffdhe2048.txt /etc/nginx/ssl-dhparams.pem:
------
failed to solve: failed to load cache key: invalid not-modified ETag: "65a0156d-1a7"

The relevent part of the compose file:

name: "OURPROJECT"

services:
  nginx-frontend:
    image: "registry.gitlab.com/OURCOMPANY/OURPROJECT/nginx-frontend:${TAG:-latest}"
    build:
      context: "./nginx-frontend/"
      args:
        TAG: "${TAG:-latest}"

      # Record metadata and Docker image labels for tracking and debugging purposes:
      labels:
        com.OURCOMPANY.OURPROJECT.gitlab_cicd.is_pipeline_build: "${GITLAB_CI:-false}"
        com.OURCOMPANY.OURPROJECT.gitlab_cicd.build_job.id: "${CI_JOB_ID}"
        com.OURCOMPANY.OURPROJECT.gitlab_cicd.pipeline.id: "${CI_PIPELINE_ID}"
        com.OURCOMPANY.OURPROJECT.gitlab_cicd.project: "${CI_PROJECT_NAME}"
        com.OURCOMPANY.OURPROJECT.gitlab_cicd.docker_tag: "${TAG:-latest}"
        com.OURCOMPANY.OURPROJECT.gitlab_cicd.git_commit.branch: "${CI_COMMIT_BRANCH}"
        com.OURCOMPANY.OURPROJECT.gitlab_cicd.git_commit.hash: "${CI_COMMIT_SHA}"
        com.OURCOMPANY.OURPROJECT.gitlab_cicd.git_commit.time: "${GIT_COMMIT_TIMESTAMP_UTC}"

    container_name: "nginx-frontend"
    hostname: "nginx-frontend.container"

    ports:
      - "${OURPROJECT_HTTP_PORT_1:-80}:443"
      - "${OURPROJECT_HTTP_PORT_2:-443}:443"

    volumes:
      # TODO: Temporary self-signed cert in repo; add creating cert to install process
      - "frontend-ssl:/etc/nginx/ssl/:ro"

    restart: "unless-stopped"
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
    mem_reservation: "256m"
  # End 'nginx-frontend'

volumes:
  # Stores the SSL certificate files
  frontend-ssl:
    name: "OURPROJECT-frontend-ssl"

And the Dockerfile:

# Latest version of tag '1.27.0-alpine' (NGINX v1.27.0 based on Alpine Linux) as of 2024-07-11:
FROM nginx@sha256:a45ee5d042aaa9e81e013f97ae40c3dda26fbe98f22b6251acdf28e579560d55

# Download DH paramaters file
ADD "https://ssl-config.mozilla.org/ffdhe2048.txt" "/etc/nginx/ssl-dhparams.pem"

# Install our configuation
COPY "./nginx.conf" "/etc/nginx/nginx.conf"

# TODO: Temporary self-signed cert in repo; add creating cert to install process
COPY "./ssl/" "/etc/nginx/ssl/"

The version of Docker I'm using:

macdjord@MYWORKMACHINE:~/source/OURPROJECT.git$ docker version
Client: Docker Engine - Community
 Version:           23.0.0
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        e92dd87
 Built:             Wed Feb  1 17:47:51 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop  ()
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Aug 27 14:15:15 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

The image from the previous, successful build:

macdjord@MYWORKMACHINE:~/source/OURPROJECT.git$ docker inspect registry.gitlab.com/OURCOMPANY/OURPROJECT/nginx-frontend
[
    {
        "Id": "sha256:49f061c019cdc5ce2cc61fb8021bc51a11fa90d595197f7e0b37255da5683bf7",
        "RepoTags": [
            "registry.gitlab.com/OURCOMPANY/OURPROJECT/nginx-frontend:latest"
        ],
        "RepoDigests": [],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2024-08-29T20:49:25.134516195Z",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.27.0",
                "PKG_RELEASE=2",
                "NJS_VERSION=0.8.4",
                "NJS_RELEASE=2"
            ],
            "Cmd": [
                "nginx",
                "-g",
                "daemon off;"
            ],
            "ArgsEscaped": true,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.project": "OURPROJECT",
                "com.docker.compose.service": "nginx-frontend",
                "com.docker.compose.version": "2.15.1",
                "com.OURCOMPANY.OURPROJECT.gitlab_cicd.build_job.id": "",
                "com.OURCOMPANY.OURPROJECT.gitlab_cicd.docker_tag": "latest",
                "com.OURCOMPANY.OURPROJECT.gitlab_cicd.git_commit.branch": "",
                "com.OURCOMPANY.OURPROJECT.gitlab_cicd.git_commit.hash": "",
                "com.OURCOMPANY.OURPROJECT.gitlab_cicd.git_commit.time": "",
                "com.OURCOMPANY.OURPROJECT.gitlab_cicd.is_pipeline_build": "false",
                "com.OURCOMPANY.OURPROJECT.gitlab_cicd.pipeline.id": "",
                "com.OURCOMPANY.OURPROJECT.gitlab_cicd.project": "",
                "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>"
            },
            "StopSignal": "SIGQUIT"
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 43250893,
        "VirtualSize": 43250893,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/tj6xlwf9y01qypwtpvoe9wq15/diff:/var/lib/docker/overlay2/nezntsqz0xlyhgsi9styvb9l4/diff:/var/lib/docker/overlay2/f01827bcc99ec62736d0de7a16b8c34bc68519e17907697b0b6d8624d993fde9/diff:/var/lib/docker/overlay2/12c08844947f4c8e458eca1477d15ed6ba3f03b48656c8fbfe507bbcd0410aac/diff:/var/lib/docker/overlay2/42adcbc58f557041f56e7262bebae26aefd3e8ee4dade92ce05a13a2920ad682/diff:/var/lib/docker/overlay2/b5e3949d1e890e7bc9e2cf2da5c6290623621864fbe71dccefd7dbc4e6a378b8/diff:/var/lib/docker/overlay2/b728b0e45620b0c1da444c8d48b4b627b1d456af2c3e1aef2624816245df2b17/diff:/var/lib/docker/overlay2/f7e9e86e6fa49842c7f174ce76bcb19dfaa055e61a245fcebdb79a85005c9dd2/diff:/var/lib/docker/overlay2/71b7be2b75f95a54d1ac55d2db667cb3c99197cc1ddc95da73881905d4e403b1/diff:/var/lib/docker/overlay2/48d3722f148dcab330cf55d5fe3cd08564790d86bfb152cd9ab0392d3b014c95/diff",
                "MergedDir": "/var/lib/docker/overlay2/z0v43z8arbx8uiscdp8ui1gai/merged",
                "UpperDir": "/var/lib/docker/overlay2/z0v43z8arbx8uiscdp8ui1gai/diff",
                "WorkDir": "/var/lib/docker/overlay2/z0v43z8arbx8uiscdp8ui1gai/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:af9a70194aa4d12f967dbd4bcb1ce9c98ba42adb4ec05536080fd4560155e809",
                "sha256:b5d2e1fcf1ad14661a34c78538bc39808a70a59c2c60047e03018e1f2aa2ee51",
                "sha256:6e92270dbfe60544e45dc91744c434b891e5e15d9a4d72a001926fd3bcbdf434",
                "sha256:4275164ce2252d044dda94eee659a413704650250c070c98738e74f65ed83c3a",
                "sha256:d2cef4a1b224541e11b61653984e987e04477f39f24d16c5467fab4752baab5f",
                "sha256:320c8baef084c4861474945b59ed403d5ece64ced568fe6e271df098a14e2117",
                "sha256:b7486fe26981fbbd5ade99ab4f0d44fe143eff6ea3f743bba91bbef5c7950abe",
                "sha256:a51b172d7184285fd8b53526eeddd5282e5c8514d6627b594a4f3eebf2d24b02",
                "sha256:98c3da3459d6258ee2ffa5c9a2d89f5bb19dcddccfd99c2a6714ad0ff1bf8c56",
                "sha256:5f94d82dcbe0b9249f8b59472ce81ed8cc10ee416e409e09e305fd368f83664e",
                "sha256:b65c7c965e95570403b3ee8344b88f94f4bfc734e0e7f4b91816043e410c295b"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]
macdjord@MYWORKMACHINE:~/source/OURPROJECT.git$ docker history registry.gitlab.com/OURCOMPANY/OURPROJECT/nginx-frontend
IMAGE          CREATED        CREATED BY                                      SIZE      COMMENT
49f061c019cd   3 weeks ago    COPY ./ssl/ /etc/nginx/ssl/ # buildkit          5.02kB    buildkit.dockerfile.v0
<missing>      3 weeks ago    COPY ./nginx.conf /etc/nginx/nginx.conf # bu…   4.53kB    buildkit.dockerfile.v0
<missing>      2 months ago   ADD https://ssl-config.mozilla.org/ffdhe2048…   423B      buildkit.dockerfile.v0
<missing>      3 months ago   RUN /bin/sh -c set -x     && apkArch="$(cat …   31.2MB    buildkit.dockerfile.v0
<missing>      3 months ago   ENV NJS_RELEASE=2                               0B        buildkit.dockerfile.v0
<missing>      3 months ago   ENV NJS_VERSION=0.8.4                           0B        buildkit.dockerfile.v0
<missing>      3 months ago   CMD ["nginx" "-g" "daemon off;"]                0B        buildkit.dockerfile.v0
<missing>      3 months ago   STOPSIGNAL SIGQUIT                              0B        buildkit.dockerfile.v0
<missing>      3 months ago   EXPOSE map[80/tcp:{}]                           0B        buildkit.dockerfile.v0
<missing>      3 months ago   ENTRYPOINT ["/docker-entrypoint.sh"]            0B        buildkit.dockerfile.v0
<missing>      3 months ago   COPY 30-tune-worker-processes.sh /docker-ent…   4.62kB    buildkit.dockerfile.v0
<missing>      3 months ago   COPY 20-envsubst-on-templates.sh /docker-ent…   3.02kB    buildkit.dockerfile.v0
<missing>      3 months ago   COPY 15-local-resolvers.envsh /docker-entryp…   336B      buildkit.dockerfile.v0
<missing>      3 months ago   COPY 10-listen-on-ipv6-by-default.sh /docker…   2.12kB    buildkit.dockerfile.v0
<missing>      3 months ago   COPY docker-entrypoint.sh / # buildkit          1.62kB    buildkit.dockerfile.v0
<missing>      3 months ago   RUN /bin/sh -c set -x     && addgroup -g 101…   4.61MB    buildkit.dockerfile.v0
<missing>      3 months ago   ENV PKG_RELEASE=2                               0B        buildkit.dockerfile.v0
<missing>      3 months ago   ENV NGINX_VERSION=1.27.0                        0B        buildkit.dockerfile.v0
<missing>      3 months ago   LABEL maintainer=NGINX Docker Maintainers <d…   0B        buildkit.dockerfile.v0
<missing>      3 months ago   /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      3 months ago   /bin/sh -c #(nop) ADD file:fb066571462e703f8…   7.4MB

The version of https://ssl-config.mozilla.org/ffdhe2048.txt currently being served is identical to that stored in the latest successful build, and the ETag currently being returned is in fact "65a0156d-1a7":

macdjord@MYWORKMACHINE:~/source/OURPROJECT.git$ docker run --rm -it registry.gitlab.com/OURCOMPANY/OURPROJECT/nginx-frontend cat '/etc/nginx/ssl-dhparams.pem' && echo ''
-----BEGIN DH PARAMETERS-----
MIIBCAKCAQEA//////////+t+FRYortKmq/cViAnPTzx2LnFg84tNpWp4TZBFGQz
+8yTnc4kmz75fS/jY2MMddj2gbICrsRhetPfHtXV/WVhJDP1H18GbtCFY2VVPe0a
87VXE15/V8k1mE8McODmi3fipona8+/och3xWKE2rec1MKzKT0g6eXq8CrGCsyT7
YdEIqUuyyOP7uWrat2DX9GgdT0Kj3jlN9K5W7edjcrsZCwenyO4KbXCeAvzhzffi
7MA0BM0oNC9hkXL+nOmFg/+OTxIy7vKBg8P+OxtMb61zO7X8vC7CIAXFjvGDfRaD
ssbzSibBsu/6iGtCOGEoXJf//////////wIBAg==
-----END DH PARAMETERS-----
macdjord@MYWORKMACHINE:~/source/OURPROJECT.git$ curl -v https://ssl-config.mozilla.org/ffdhe2048.txt >'ssl-dhparams.pem'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 185.199.108.153:443...
* Connected to ssl-config.mozilla.org (185.199.108.153) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS header, Certificate Status (22):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.2 (IN), TLS header, Finished (20):
{ [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2587 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.2 (OUT), TLS header, Finished (20):
} [5 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=ssl-config.mozilla.org
*  start date: Sep  4 21:18:30 2024 GMT
*  expire date: Dec  3 21:18:29 2024 GMT
*  subjectAltName: host "ssl-config.mozilla.org" matched cert's "ssl-config.mozilla.org"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55dcda78d550)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
> GET /ffdhe2048.txt HTTP/2
> Host: ssl-config.mozilla.org
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [193 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
< HTTP/2 200
< server: GitHub.com
< content-type: text/plain; charset=utf-8
< last-modified: Thu, 11 Jan 2024 16:21:01 GMT
< access-control-allow-origin: *
< etag: "65a0156d-1a7"
< expires: Fri, 20 Sep 2024 17:39:38 GMT
< cache-control: max-age=600
< x-proxy-cache: MISS
< x-github-request-id: E8BD:3CEE54:DDC5D9:F801F2:66EDB102
< accept-ranges: bytes
< age: 0
< date: Fri, 20 Sep 2024 18:51:54 GMT
< via: 1.1 varnish
< x-served-by: cache-yul1970030-YUL
< x-cache: HIT
< x-cache-hits: 0
< x-timer: S1726858314.084402,VS0,VE22
< vary: Accept-Encoding
< x-fastly-request-id: 8dfe1114d2a9c8deb9f31f15c6a4e43fc1c36016
< content-length: 423
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
100   423  100   423    0     0   1861      0 --:--:-- --:--:-- --:--:--  1871
* Connection #0 to host ssl-config.mozilla.org left intact
macdjord@MYWORKMACHINE:~/source/OURPROJECT.git$ cat 'ssl-dhparams.pem' && echo ''
-----BEGIN DH PARAMETERS-----
MIIBCAKCAQEA//////////+t+FRYortKmq/cViAnPTzx2LnFg84tNpWp4TZBFGQz
+8yTnc4kmz75fS/jY2MMddj2gbICrsRhetPfHtXV/WVhJDP1H18GbtCFY2VVPe0a
87VXE15/V8k1mE8McODmi3fipona8+/och3xWKE2rec1MKzKT0g6eXq8CrGCsyT7
YdEIqUuyyOP7uWrat2DX9GgdT0Kj3jlN9K5W7edjcrsZCwenyO4KbXCeAvzhzffi
7MA0BM0oNC9hkXL+nOmFg/+OTxIy7vKBg8P+OxtMb61zO7X8vC7CIAXFjvGDfRaD
ssbzSibBsu/6iGtCOGEoXJf//////////wIBAg==
-----END DH PARAMETERS-----

(I'd like to check what the ETag stored in the build cache is, but I can't find any command that will show me this information.)

Related issues: #905 and #2832 both describe the same issue. Both are marked as closed.

Further testing: I can delete the existing image and rebuild from scratch, without the cached build. I've held off in case there is other information to be gleaned from the old image before I delete it (such as what ETag it expects for that ADD command).

[macdjord]

Just to be clear here: I'd really like to inspect the build cache metadata for the ADD https://ssl-config.mozilla.org/ffdhe2048.txt layer of the existing image, so that I can check things like 'what ETag is it expecting?', but I don't know a command that will let me do that. If anyone can tell me such a command, I'd appreciate it.

[tonistiigi]

The error means that:

• You made a successful build. Server responded content with etag that was saved (I don't think that etag is visible in the output, lets say it was "xyz")
• New build is making a HEAD request with If-None-Match: xyz (previous etag, there can also be multiple such etags if you have done many builds and got different results from the URL).
• The server responds 304, indicating that previous data is still valid. That request contains Etag ("65a0156d-1a7"), but it doesn't match any of the etags that were in the request. That confuses buildkit as it doesn't know what previous reference the server considered as matching.

Because some servers misbehave we allow a case where 304 doesn't contain etag and request only contained one to consider that the one we sent was matching. But in this case server does seem to send etag but it is not the correct one that should cause 304.

You can't see the raw etag metadata via API. What you can do is find the previous cache record from buildx du --verbose via the description and then prune it with --filter id=value filter.