Something's wrong with the dockerhub setup

[EinarElen]

Describe the bug After having a succesful build and upload of #65 I tried running

ldmx use dev iss65-split-packages
Digest: sha256:ceea385c30998e97d04d44fa681584c942e3a5f88895308c84eca430f85e2313
Status: Downloaded newer image for ldmx/dev:iss65-split-packages

Doing ldmx pull ... make no difference

LDMX config gives

LDMX base directory: /home/einarelen/ldmx
uname: Linux elfriede 5.15.0-78-generic #85~20.04.1-Ubuntu SMP Mon Jul 17 09:42:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
OSTYPE: linux-gnu
Bash version: 5.0.17(1)-release
Display Port: 
Container Mounts: /home/einarelen/ldmx
Container Environments: 
Docker Version: Docker version 24.0.5, build ced0996
Docker Tag: ldmx/dev:iss65-split-packages
    Digest: ldmx/dev@sha256:006a2d7d7936010522ad122edf0430f68e214b6a6ff3c1e702294010fe38184c

Note: The digest hash here doesn't match the one from above, not sure if that's relevant.

Trying to run any of the new packages that was added in the build doesn't work either directly or from the hash from ldmx config or from the ldmx use output

> einarelen@elfriede:~/ldmx/ldmx-sw/build_debug$ ldmx ccmake 
/etc/entry.sh: line 55: ccmake: command not found
> einarelen@elfriede:~/ldmx/ldmx-sw/build_debug$ docker run --rm -it ldmx/dev@sha256:006a2d7d7936010522ad122edf0430f68e214b6a6ff3c1e702294010fe38184c . bash 
> root@d6fdad158f5c:/# ccmake
bash: ccmake: command not found
>einarelen@elfriede:~/ldmx/ldmx-sw/build_debug$ docker run --rm -it ldmx/dev@sha256:ceea385c30998e97d04d44fa681584c942e3a5f88895308c84eca430f85e2313 . ccmake 
/etc/entry.sh: line 55: ccmake: command not found

If I go to dockerhub https://hub.docker.com/layers/ldmx/dev/iss65-split-packages/images/sha256-cbbd9e57d8aa29417a269af72cc37fac5ee184094b36ca66a796e0d64f86c4b2?context=explore for the push, the hash in the URL is for the ARM build. Is the issue that we are trying to run this and fall back to an older version when it doesn't work?

If you go to the tag itself and pick the most recent layer with AMD

>  docker run --rm -it ldmx/dev@sha256:a06b1c992ca7169e04c0998a8a6e46f58ae3c3b44051a26c3251cb9883896408 . ccmake 
... 
Digest: sha256:a06b1c992ca7169e04c0998a8a6e46f58ae3c3b44051a26c3251cb9883896408
Status: Downloaded newer image for ldmx/dev@sha256:a06b1c992ca7169e04c0998a8a6e46f58ae3c3b44051a26c3251cb9883896408
Usage

  ccmake <path-to-source>
  ccmake <path-to-existing-build>

Specify a source directory to (re-)generate a build system for it in the
current working directory.  Specify an existing build directory to
re-generate its build system.

Run 'ccmake --help' for more information.

So that works!

[tomeichlersmith]

:facepalm: If you go to the "Digests" artifact of the last run you made and unzip it, it will contain empty files whose names are the digests of the images that would be put together into the manifest.

There are like a dozen digests. There should be two. I think what's happening is that I am not cleaning up the /tmp directory and so it is including all of the digests from previous builds.

[tomeichlersmith]

https://github.com/LDMX-Software/docker/blob/739e1e7e06789808bb1c7ca6aa779f8e575cb336/.github/workflows/ci.yml#L63-L75

I'll need to figure out a creative solution for cleaning up the /tmp/digests directory. In the previous GitHub-runner, the /tmp was cleared by the host after every job while in this setup, we are not clearing the runner since we want the cache to be kept around. I'm thinking I could update the artifact mechanism to upload the digests themselves and then use the download-all mechanism to get the two that were built during the merge step.

[tomeichlersmith]

@EinarElen I can log into the runner tomorrow and manually make the manifest for you so that you can have a functional build of the branch you want to test.

[EinarElen]

That's fine, I could test it enough for my purposes with the workaround!

[tomeichlersmith]

Alright, I manually fixed up the images pushed to docker hub (notes on how I did it are below). Testing a redesign of the digest-passing in the workflows on the connected branch.

How

I manually looked at the run logs to get the two digests corresponding to the two different images built (one for each architecture) and the image tags that should be grouped with them. Then I used buildx imagetools to remake the manifest as is done in the workflow. For example,

docker buildx imagetools create -t ldmx/dev:main \ 
  ldmx/dev@sha256:ab9ef47d1d9472f33cb5d0b9f1f1731e412af16daa2b09d9c50e16f322b3a45b \
  ldmx/dev@sha256:745f76543b1b3044456d36154be96f3d09ca03f74c5519b2716a7db57b5b0d15

This only worked after I logged in to dockerhub (before that, I got a "permission denied" style error) and so I only think admins of the ldmx group on docker hub would be able to run this command.

[tomeichlersmith]

Fix Validation

tom@appa:~/code$ docker run ldmx/dev:iss65-split-packages . ccmake
Unable to find image 'ldmx/dev:iss65-split-packages' locally
iss65-split-packages: Pulling from ldmx/dev
b237fe92c417: Pull complete 
3256e785a7ec: Pull complete 
c2d2a234bda0: Pull complete 
b918729b3726: Pull complete 
668f546d29d8: Pull complete 
9720fd3c32c8: Pull complete 
5fb45e5366f8: Pull complete 
cfdbf6a36043: Pull complete 
27378556c612: Pull complete 
0e5d957be2a2: Pull complete 
dbe517e1044c: Pull complete 
a491457ae881: Pull complete 
4d46a6032b56: Pull complete 
355a6f44baa5: Pull complete 
a067fafb8dd8: Pull complete 
1c1596eef39c: Pull complete 
21fa50e560e3: Pull complete 
06a9377eefde: Pull complete 
56dc5f3cb7d3: Pull complete 
1b9afab32bb0: Pull complete 
58998cee308d: Pull complete 
74c668a3573f: Pull complete 
4ea1c54a63e8: Pull complete 
b6745abf528c: Pull complete 
61cbc7ee2a5f: Pull complete 
c0a7f2889863: Pull complete 
Digest: sha256:0a7005dd6e497a953972fc12961447355522f4740fbbea513a34023184840ff9
Status: Downloaded newer image for ldmx/dev:iss65-split-packages
Usage

  ccmake <path-to-source>
  ccmake <path-to-existing-build>

Specify a source directory to (re-)generate a build system for it in the
current working directory.  Specify an existing build directory to
re-generate its build system.

Run 'ccmake --help' for more information.

[EinarElen]

Works like a charm :)