Docker ckan-dev tags 2.9/2.10 failing since latest tags pushed (2.10 - 24c3221637f0, 2.9 - 095615410276 )

[duttonw]

Something outside of this repo's version control has occurred and now we have failing cicd builds:

Our cicd jobs extend the image and is now having issues which it did not have yesterday. Nothing has been changed in our repo/workflows and I was able to retrigger it on a passing run which is less than 24hours old.

Passing (Triggered via push 18 hours ago) : https://github.com/qld-gov-au/ckanext-ytp-comments/actions/runs/12005390429/attempts/1

5 [ckan 1/10] FROM docker.io/ckan/ckan-dev:2.10@sha256:4f8f8a27f72ddc4509e3fc1c8c29b6615b205d04d12922019629e98acbec124c

Failing: https://github.com/qld-gov-au/ckanext-ytp-comments/actions/runs/12005390429

5 [ckan 1/10] FROM docker.io/ckan/ckan-dev:2.10@sha256:4117dd652dac492431d0c932642c5724882294ea042ec782dc7909ec96f4c45a

2.10

#6 [ckan  2/10] RUN mkdir /docker-entrypoint.d
#6 0.443 mkdir: can't create directory '/docker-entrypoint.d': File exists
time="2024-11-26T02:02:10Z" level=warning msg="current commit information was not captured by the build" error="failed to read current commit information with git rev-parse --is-inside-work-tree"
failed to solve: process "/bin/sh -c mkdir /docker-entrypoint.d" did not complete successfully: exit code: 1
#6 ERROR: process "/bin/sh -c mkdir /docker-entrypoint.d" did not complete successfully: exit code: 1
------
 > [ckan  2/10] RUN mkdir /docker-entrypoint.d:
0.443 mkdir: can't create directory '/docker-entrypoint.d': File exists
------

2.11

#6 [ckan  2/10] RUN mkdir /docker-entrypoint.d
#6 0.179 mkdir: can't create directory '/docker-entrypoint.d': File exists
time="2024-11-26T02:02:09Z" level=warning msg="current commit information was not captured by the build" error="failed to read current commit information with git rev-parse --is-inside-work-tree"
failed to solve: process "/bin/sh -c mkdir /docker-entrypoint.d" did not complete successfully: exit code: 1
#6 ERROR: process "/bin/sh -c mkdir /docker-entrypoint.d" did not complete successfully: exit code: 1
------
 > [ckan  2/10] RUN mkdir /docker-entrypoint.d:
0.179 mkdir: can't create directory '/docker-entrypoint.d': File exists
------

Below is the current point in time details of the tag's and hashes as of this momenet this ticket is created https://hub.docker.com/r/ckan/ckan-dev/tags

TAG

2.9-v20241125

Last pushed 11 hours ago by amercader

docker pull ckan/ckan-dev:2.9-v20241125

Copy

Digest | OS/ARCH | Compressed size -- | -- | -- 095615410276 | linux/amd64 | 602.36 MB

TAG [2.9-v20241125](https://hub.docker.com/layers/ckan/ckan-dev/2.9-v20241125/images/sha256-095615410276c5b311f2888584e73b569754c40366fdf3ae23d41172a13a0ee3?context=explore) Last pushed 11 hours ago by [amercader](https://hub.docker.com/u/amercader) docker pull ckan/ckan-dev:2.9-v20241125 Digest OS/ARCH Compressed size [095615410276](https://hub.docker.com/layers/ckan/ckan-dev/2.9-v20241125/images/sha256-095615410276c5b311f2888584e73b569754c40366fdf3ae23d41172a13a0ee3?context=explore) linux/amd64 602.36 MB TAG [2.9.11](https://hub.docker.com/layers/ckan/ckan-dev/2.9.11/images/sha256-095615410276c5b311f2888584e73b569754c40366fdf3ae23d41172a13a0ee3?context=explore) Last pushed 11 hours ago by [amercader](https://hub.docker.com/u/amercader) docker pull ckan/ckan-dev:2.9.11 Digest OS/ARCH Compressed size [095615410276](https://hub.docker.com/layers/ckan/ckan-dev/2.9.11/images/sha256-095615410276c5b311f2888584e73b569754c40366fdf3ae23d41172a13a0ee3?context=explore) linux/amd64 602.36 MB TAG [2.9](https://hub.docker.com/layers/ckan/ckan-dev/2.9/images/sha256-095615410276c5b311f2888584e73b569754c40366fdf3ae23d41172a13a0ee3?context=explore) Last pushed 11 hours ago by [amercader](https://hub.docker.com/u/amercader) docker pull ckan/ckan-dev:2.9 Digest OS/ARCH Compressed size [095615410276](https://hub.docker.com/layers/ckan/ckan-dev/2.9/images/sha256-095615410276c5b311f2888584e73b569754c40366fdf3ae23d41172a13a0ee3?context=explore) linux/amd64 602.36 MB TAG [2.10-v20241125](https://hub.docker.com/layers/ckan/ckan-dev/2.10-v20241125/images/sha256-24c3221637f01c874050db770dabb8ad0bbbf70355350669d30bb29f47031155?context=explore) Last pushed 11 hours ago by [amercader](https://hub.docker.com/u/amercader) docker pull ckan/ckan-dev:2.10-v20241125 Digest OS/ARCH Compressed size [24c3221637f0](https://hub.docker.com/layers/ckan/ckan-dev/2.10-v20241125/images/sha256-24c3221637f01c874050db770dabb8ad0bbbf70355350669d30bb29f47031155?context=explore) linux/amd64 704.11 MB TAG [2.10.5](https://hub.docker.com/layers/ckan/ckan-dev/2.10.5/images/sha256-24c3221637f01c874050db770dabb8ad0bbbf70355350669d30bb29f47031155?context=explore) Last pushed 11 hours ago by [amercader](https://hub.docker.com/u/amercader) docker pull ckan/ckan-dev:2.10.5 Digest OS/ARCH Compressed size [24c3221637f0](https://hub.docker.com/layers/ckan/ckan-dev/2.10.5/images/sha256-24c3221637f01c874050db770dabb8ad0bbbf70355350669d30bb29f47031155?context=explore) linux/amd64 704.11 MB TAG [2.10](https://hub.docker.com/layers/ckan/ckan-dev/2.10/images/sha256-24c3221637f01c874050db770dabb8ad0bbbf70355350669d30bb29f47031155?context=explore) Last pushed 11 hours ago by [amercader](https://hub.docker.com/u/amercader) docker pull ckan/ckan-dev:2.10 Digest OS/ARCH Compressed size [24c3221637f0](https://hub.docker.com/layers/ckan/ckan-dev/2.10/images/sha256-24c3221637f01c874050db770dabb8ad0bbbf70355350669d30bb29f47031155?context=explore) linux/amd64 704.11 MB

[wardi]

I do see the directory /docker-entrypoint.d/ in ckan/ckan-dev:2.9-v20241125, even though the mkdir command is an https://github.com/ckan/ckan-docker-base/blob/main/ckan-2.9/Dockerfile#L126 ONBUILD RUN command which IIUC is a trigger that is only supposed to be run by downstream images.

I'm not sure why we didn't create this directory with a normal RUN command and make it part of the image instead of using this ONBUILD trigger feature but it seems that both are happening for some reason and causing this error.

So we can either:

1. figure out why the image is now being created with the /docker-entrypoint.d/ directory ( @amercader could this be related to using DOCKER_BUILDKIT=1 now?)
2. stop using ONBUILD, or
3. just use mkdir -p like https://github.com/ckan/ckan-docker-base/pull/94

What do you think @duttonw ?

[amercader]

I bet is this buildkit bug: https://github.com/moby/buildkit/issues/5486

Any of the options you mention sounds good to me

[wardi]

I like option 2. If removing ONBUILD fixes the issue then that gets us a simpler Dockerfile at the same time.

[duttonw]

its all magic sauce and the use here does not fit the normal usecase given in the docs. https://docs.docker.com/reference/dockerfile/#onbuild

I'd say remove ONBUILD, but still leave in mkdir -p for safety.