Open MAHDTech opened 5 months ago
Looks like #97 has the same symptom
@MAHDTech Could you try by removing all copy-args
(at least all non strictly required to push the image)?
Hey @nlewo,
Yeah I've tried where the only args are the dest creds and still has the same error.
- id: container
name: Build container
run: >
devenv container
--registry docker://${{ env.REGISTRY }}/${{ env.PROJECT }}/
--copy-args="--dest-creds=${{ github.actor }}:${{ secrets.GITHUB_TOKEN }}"
copy ${{ matrix.container }}
Results in;
Copying blob sha256:fd8a0cc19f3537e316cce2fe05818f25ea0a89e64dffd5621aabc8503b36f894
time="2024-04-03T08:59:58Z" level=fatal msg="writing blob: Patch \"[https://ghcr.io/v2/***/containers/test/blobs/upload/e51cd0a5-7676-4187-b9af-70a4d451bd23\](https://ghcr.io/v2/***/containers/test/blobs/upload/e51cd0a5-7676-4187-b9af-70a4d451bd23/)": happened during read: Digest did not match, expected sha256:f0c636e0ef473fd1bb68a3bade4e24bdc202a0b86e05702bdad7c6f5453b26dd, got sha256:a908ca128957816d3b326cb489afe5db6d5bf1071113b0d983e3651d06b1efd3"
✔ Copying test container in 11.9s.
Error: × Failed to copy container
Well, this kind of issue are currently pretty painful to debug because you need to compare how the layer hash is computed at build time to the layer hash computed at runtime.
In nix2container, there is the option buildImage.reproducible
to avoid such issues (but this implies degraded performances). This option doesn't seem exposed in devenv.
Are you able to reproduce locally?
Root cause: https://github.com/NixOS/nix/issues/10525
@Atry Thank you and congrats for the debugging!
(I would really like to write a tool generating the list of files (attribute + hash) at build time and runtime in order to compare them easily, but not enough time on my spare time.)
I encountered the same error message when trying to use devenv container run
, and still saw the same error after updating Nix to 2.22.0 (which I think includes Atry's fix NixOS/nix#10456).
$ nix --version
nix (Nix) 2.22.0
A small reproducible example, which uses forked devenv and nix2container to be able to pass reproducible
to buildImage
: https://github.com/cachix/devenv/compare/main...ento:devenv:container-digest-mismatch-repro
Patches in the forked versions make it so that reproducible
flag set in devenv.nix
gets passed onto both the layer built by devenv and the customization layer built by nix2container. The latter change was needed to make the digest mismatch error go away.
git clone https://github.com/ento/devenv
cd devenv
git checkout container-digest-mismatch-repro
cd examples/container
direnv allow
# container that sets reproducible=false
devenv container run good
# container that sets reproducible=true
devenv container run bad
```sh • Building bad container ... • Using Cachix: devenv warning: creating lock file '/examples/container/devenv.lock' warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled /nix/store/5vpy61qxm85izcx8bnhs24g5xlwzvrsg-image-bad.json ✔ Building bad container in 9.8s. warning: creating lock file '/examples/container/devenv.lock' • Running /nix/store/0rrxkx0lj6agbc72imvs5hkyyrv64xcy-copy-container /nix/store/5vpy61qxm85izcx8bnhs24g5xlwzvrsg-image-bad.json docker-daemon: Copying container /nix/store/5vpy61qxm85izcx8bnhs24g5xlwzvrsg-image-bad.json to docker-daemon:bad:latest Getting image source signatures Copying blob db3b02af90da done | Copying blob c99a1b2b3ae9 [=====================================>] 391.2MiB / 391.2MiB | 30.5 MiB/s FATA[0001] writing blob: writing to temporary on-disk layer: happened during read: Digest did not match, expected sha256:c99a1b2b3ae9337c455c3ffe47eb8b54f5209b07e9b26261cd15a083bbcb755a, got sha256:656d7d142bca6908cb75c40e2f1150e3ae7855d28a4fb0a18268cc40a6f2e359 ✔ Copying bad container in 2.7s. Error: × Failed to copy container ```
```sh $ devenv container run good • Building good container ... • Using Cachix: devenv warning: creating lock file '/examples/container/devenv.lock' /nix/store/7q0icf7bw0ngqfjmh5kxnnmgdb0zy8ag-image-good.json ✔ Building good container in 1.5s. warning: creating lock file '/examples/container/devenv.lock' • Running /nix/store/k4hl27rchhpkhrq76278kq3pphpgz158-copy-container /nix/store/7q0icf7bw0ngqfjmh5kxnnmgdb0zy8ag-image-good.json docker-daemon: Copying container /nix/store/7q0icf7bw0ngqfjmh5kxnnmgdb0zy8ag-image-good.json to docker-daemon:good:latest Getting image source signatures Copying blob db3b02af90da done | Copying blob c99a1b2b3ae9 done | Copying config dd21b4c201 done | Writing manifest to image destination ✔ Copying good container in 2.5s. warning: creating lock file '/examples/container/devenv.lock' mkdir: cannot create directory '/run': Permission denied ✔ Running good container in 1.3s. ```
This gives me an image JSON file with layers pointing to tarballs, which I guess are build-time artifacts. I'm not sure how to get ahold of runtime blobs to compare against.
$ grep layer-path /nix/store/7q0icf7bw0ngqfjmh5kxnnmgdb0zy8ag-image-good.json
"layer-path": "/nix/store/ff1qzsi9s0raps1375fal4cvc5nffrlj-layers.json/db3b02af90da10c12c8657cfc05fa9cfe2bb0b0b22b7d166bb049d5b63ae4e85.tar"
"layer-path": "/nix/store/h2aa9mp4s1x0w11j5ynra4yf1idphhjs-layers.json/c99a1b2b3ae9337c455c3ffe47eb8b54f5209b07e9b26261cd15a083bbcb755a.tar"
$ sha256sum /nix/store/ff1qzsi9s0raps1375fal4cvc5nffrlj-layers.json/db3b02af90da10c12c8657cfc05fa9cfe2bb0b0b22b7d166bb049d5b63ae4e85.tar
db3b02af90da10c12c8657cfc05fa9cfe2bb0b0b22b7d166bb049d5b63ae4e85 /nix/store/ff1qzsi9s0raps1375fal4cvc5nffrlj-layers.json/db3b02af90da10c12c8657cfc05fa9cfe2bb0b0b22b7d166bb049d5b63ae4e85.tar
$ sha256sum /nix/store/h2aa9mp4s1x0w11j5ynra4yf1idphhjs-layers.json/c99a1b2b3ae9337c455c3ffe47eb8b54f5209b07e9b26261cd15a083bbcb755a.tar
c99a1b2b3ae9337c455c3ffe47eb8b54f5209b07e9b26261cd15a083bbcb755a /nix/store/h2aa9mp4s1x0w11j5ynra4yf1idphhjs-layers.json/c99a1b2b3ae9337c455c3ffe47eb8b54f5209b07e9b26261cd15a083bbcb755a.tar
Can you check what version of nix-daemon is running with nix store ping
Ohh. v2.22.0.
$ nix store ping
warning: 'nix store ping' is a deprecated alias for 'nix store info'
Store URL: local
Version: 2.22.0
Trusted: 1
This gave me hope because I'd rebooted my machine after posting my previous comment (so the daemon may not have been on the updated version then), but no, it still errors with the same message and hashes :/
$ devenv container run bad
...
FATA[0003] writing blob: writing to temporary on-disk layer: happened during read: Digest did not match, expected sha256:c99a1b2b3ae9337c455c3ffe47eb8b54f5209b07e9b26261cd15a083bbcb755a, got sha256:656d7d142bca6908cb75c40e2f1150e3ae7855d28a4fb0a18268cc40a6f2e359
I have not been able to reproduce:
devenv container run bad
• Building bad container ...
• Using Cachix: devenv
warning: creating lock file '/examples/container/devenv.lock'
/nix/store/dnw6km9h056s02183jx9ybq806xmz5s2-image-bad.json
✔ Building bad container in 21.6s.
warning: creating lock file '/examples/container/devenv.lock'
[1/1/3 built, 37 copied (301.0/301.1 MiB), 56.8 MiB DL] building skopeo-1.15.0 (buildPhase): go build "-buildmode=pie" -ldflags '-X main.gitCommit= ' -gcflags "" -tags "
• Running /nix/store/0rrxkx0lj6agbc72imvs5hkyyrv64xcy-copy-container /nix/store/dnw6km9h056s02183jx9ybq806xmz5s2-image-bad.json docker-daemon:
Copying container /nix/store/dnw6km9h056s02183jx9ybq806xmz5s2-image-bad.json to docker-daemon:bad:latest
Getting image source signatures
Copying blob db3b02af90da done |
Copying blob dc5e8b8a42d6 done |
Copying config 1eb00a101a done |
Writing manifest to image destination
✔ Copying bad container in 154.0s.
warning: creating lock file '/examples/container/devenv.lock'
✨ devenv 1.0.5 is newer than devenv input in devenv.lock. Run `devenv update` to sync.
mkdir: cannot create directory '/run': Permission denied
✔ Running bad container in 3.5s.
direnv: loading ~/tmp/devenv/examples/container/.envrc
direnv: loading https://raw.githubusercontent.com/cachix/devenv/95f329d49a8a5289d31e0982652f7058a189bfca/direnvrc (sha256-d+8cBpDfDBj41inrADaJt+bDWhOktwslgoP5YiGJ1v0=)
direnv: using devenv
direnv: ./devenv.lock changed, reloading
• Using Cachix: devenv
warning: creating lock file '/examples/container/devenv.lock'
direnv: updated devenv shell cache
✨ devenv 1.0.5 is newer than devenv input in devenv.lock. Run `devenv update` to sync.
direnv: export +DEVENV_DOTFILE +DEVENV_PROFILE +DEVENV_ROOT +DEVENV_RUNTIME +DEVENV_STATE -NIX_BUILD_CORES -__structuredAttrs -buildInputs -builder -depsBuildBuild -depsBuildBuildPropagated -depsBuildTarget -depsBuildTargetPropagated -depsHostHost -depsHostHostPropagated -depsTargetTarget -depsTargetTargetPropagated -doCheck -doInstallCheck -dontAddDisableDepTrack -nativeBuildInputs -out -outputs -patches -propagatedBuildInputs -propagatedNativeBuildInputs -shell -stdenv -strictDeps ~NIX_CFLAGS_COMPILE ~NIX_LDFLAGS ~PATH ~XDG_DATA_DIRS ~name
@ento I'm not sure, but it could be possible that the above nix issue poisoned your nix store. (Maybe you are reusing previously built store paths that doesn't have the correct hash because of the symlink issue.)
@ento Could you clear your Nix store to be sure all store paths are built with the fixed Nix version?
Is there a good way of clearning just the paths that'd need to be cleared? If not, I'll be happy to clear the entire store, but it'd be useful for others if there's a way to delete paths in a targeted fashion.
I tried deleting paths selectively like this, and the issue persisted, although the digest value changed. The same after running nix store gc
.
image_path=/nix/store/bipgbyyrjhzl13lfh8jaajdm02kqsmhz-image-bad.json
deriver=$(nix-store --query --deriver $image_path)
nix-store -q --requisites --include-outputs $image_path > drv-closure.txt
nix-store -q --requisites --include-outputs $deriver >> drv-closure.txt
tac drv-closure.txt | xargs -L1 nix-store --delete
This might be a red herring and might be expected, but digest values seem to change every time I run nix store gc
and then devenv container run bad
.
FATA[0002] writing blob: writing to temporary on-disk layer: happened during read: Digest did not match, expected sha256:3bc96411ff7cc18cd333e7e2b391d2416ca2386d91747c20ed7cf b702ef3ae40, got sha256:356d7f2c1d06023d7259b16f6c3086c5804e0d4714b5262952182e6c082bb74b ... FATA[0002] writing blob: writing to temporary on-disk layer: happened during read: Digest did not match, expected sha256:e84bb3922a353e5b5025af6e96760d8209cc155b68fdce5f458ed 7bb760156c4, got sha256:8b6bdcfc94b5e235d2712d0ed81f16d7e26c97da4cc2b95d34ae508f8c68fcf7
``` --- ../7kjzdjzq4dj8xczijnd1mmr3hvx41hc2-image-bad.json 2024-05-02 11:44:23.750682319 -0800 +++ ../sdk3vyy8vxvj5d0vrwr40z3r3kw7f848-image-bad.json 2024-05-02 11:45:58.890979017 -0800 @@ -26,10 +26,10 @@ "diff_ids": "sha256:db3b02af90da10c12c8657cfc05fa9cfe2bb0b0b22b7d166bb049d5b63ae4e85", "paths": [ { - "path": "/nix/store/gnn0qdyp7g5jax9vxkbahsmhddn59sp2-devenv-container-home", + "path": "/nix/store/zq828gjkb1vpx29m1dq4bna23s0xy7n4-devenv-container-home", "options": { "rewrite": { - "regex": "^/nix/store/gnn0qdyp7g5jax9vxkbahsmhddn59sp2-devenv-container-home", + "regex": "^/nix/store/zq828gjkb1vpx29m1dq4bna23s0xy7n4-devenv-container-home", "repl": "" }, "perms": [ @@ -48,9 +48,9 @@ "mediatype": "application/vnd.oci.image.layer.v1.tar" }, { - "digest": "sha256:0b12ef91ce5511d31a0259abc5a5675226a6327567e3b07867c70717d1e71673", + "digest": "sha256:3bc96411ff7cc18cd333e7e2b391d2416ca2386d91747c20ed7cfb702ef3ae40", "size": 410213888, - "diff_ids": "sha256:0b12ef91ce5511d31a0259abc5a5675226a6327567e3b07867c70717d1e71673", + "diff_ids": "sha256:3bc96411ff7cc18cd333e7e2b391d2416ca2386d91747c20ed7cfb702ef3ae40", "paths": [ { "path": "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc" @@ -149,10 +149,10 @@ "path": "/nix/store/k491acz8rcfhrpivv0rg6q83a14zss2j-linux-pam-1.6.0" }, { - "path": "/nix/store/g1blmfb2qsy6wk811vr04hzw90c0asgy-xz-5.4.6" + "path": "/nix/store/scc6p052526m5cky3p74d4749bpfjml3-pkg-config-0.29.2" }, { - "path": "/nix/store/scc6p052526m5cky3p74d4749bpfjml3-pkg-config-0.29.2" + "path": "/nix/store/g1blmfb2qsy6wk811vr04hzw90c0asgy-xz-5.4.6" }, { "path": "/nix/store/b5lblag9apih8iwbndwayf47ng6d6r78-ed-1.20.1" @@ -311,6 +311,9 @@ "path": "/nix/store/3nw836p7bjg3byq3ss8r8xkj96ljd8w6-envsubst-1.4.2" }, { + "path": "/nix/store/zq828gjkb1vpx29m1dq4bna23s0xy7n4-devenv-container-home" + }, + { "path": "/nix/store/x5d3vyn39k0h051i1sr41k725n6fak40-devenv-container-tmp", "options": { "rewrite": { @@ -339,9 +342,6 @@ } }, { - "path": "/nix/store/gnn0qdyp7g5jax9vxkbahsmhddn59sp2-devenv-container-home" - }, - { "path": "/nix/store/f6r9a09q13xlixby4ir3jmfw9d11gzb4-devenv-container-etc", "options": { "rewrite": { @@ -354,10 +354,10 @@ "path": "/nix/store/cnyz26jyydpccbwrm9g3v37g3zrky8d2-entrypoint" }, { - "path": "/nix/store/njlj6r93p21f7bwkmiszys91f4i2liyk-nix-database", + "path": "/nix/store/5adbaq768p4v17rm3cv234x5vb81yjx6-nix-database", "options": { "rewrite": { - "regex": "^/nix/store/njlj6r93p21f7bwkmiszys91f4i2liyk-nix-database", + "regex": "^/nix/store/5adbaq768p4v17rm3cv234x5vb81yjx6-nix-database", "repl": "" }, "perms": [ ```
This is how I delete the cache: https://github.com/Atry/nix-regression-sandbox-symbolic-links/blob/618183b5eb0c7b2687e9e699087d889216de3591/flake.nix#L23
@ento I started working a pretty useful debug option: buildImage.verifyTrace
!
It basically generates traces of all stuffs added in the image tar stream. This would allow you to diff them to identify precisely differences.
To debug your issue, you could checkout this MR: https://github.com/nlewo/nix2container/pull/135
The MR description and an added section provide some help on how to use this option.
Hi, I'm using
devenv
to build a container image withnix2container
and after the image is successfully created the following error message is shown when skopeo is uploading the image.Currently I'm using this setup
I've tried passing different copy flags to skopeo with the same result, including with and without
--preserve-digests
Anyone seen this before or have any tips where to look?