Image pull fails

[phunni]

When trying to pull an image I get the following error:

# darch images pull phunni/moses
pulling phunni/moses:latest
INFO[0262] apply failure, attempting cleanup             error="failed to extract layer sha256:6a451abf52f366ab32ffc0ff6684c6b261803ebe6b2a1c1b0a1fd22325bc7d85: failed to convert whiteout file "usr/lib/.wh.libarchive.so.13.4.0": operation not permitted: unknown" key="extract-393300867-ZU8W sha256:5e84eac8d4e28fb9773f8b0656013bf216612a14cff2eba700fad216ea917567"
darch: failed to unpack image on snapshotter overlayfs: failed to extract layer sha256:6a451abf52f366ab32ffc0ff6684c6b261803ebe6b2a1c1b0a1fd22325bc7d85: failed to convert whiteout file "usr/lib/.wh.libarchive.so.13.4.0": operation not permitted: unknown

Initially this was when I tried to pull from a local docker instance on my network. In order to try and rule out that being the problem, I then pushed this image to dockerhub and still got this error. This is on a machine already running a darch image.

Edit: Interestingly, the image pull succeeds when I'm booted into a standard (i.e. not darch) system. I've so far only tried it from dockerhub, but I'll test it from my local registry too - although I can't see why that would fail...

[aaroncwhite]

I'm seeing a similar error when I have the containerd folder mounted with an overlay. What's your partition map look like?

https://github.com/containerd/containerd/issues/2402

[aaroncwhite]

Confirmed for my setup anyways. I have a zfs pool setup to overlay /home, /var/lib/containerd, /var/lib/darch, etc to avoid managing multiple partitions.

Per the linked issue, pulling directly with sudo ctr images pull ... --snapshotter=native works; however, darch pulling still fails even if I try to set the default snapshotter behavior with cri settings in the containerd/config.toml. I moved the containerd root directory to a non-overlay folder and darch is able to run builds but fails when trying to finish a stage upload.

Moving both folders off the overlay seems to have solved the issue for now, but I'd love to have the option in the future vs managing multiple separate partitions.

It looks like darch needs a flag to set this line https://github.com/godarch/darch/blob/develop/pkg/repository/build.go#L125 (?). Based on my read, it seems like that's where it's happening, but @pauldotknopf is the expert here.

[pauldotknopf]

IMO, it's best to have /var/lib/darch be a different partition. I also configure containerd to use everything in that partition. See here.