Open amurzeau opened 1 year ago
tonistiigi
commented
1 year ago cc @sipsma
I think this is expected. When --link makes a merge base, it creates hardlinks between files that are the same in the source and merged layers. If you use a spanshotter that is not capable of doing hardlinks then it needs to do a full copy.
amurzeau
commented
1 year ago In the 1.16GB layer in fuse-overlayfs case, there are these files:
$ ls /home/doc/.local/share/buildkit/runc-fuse-overlayfs/snapshots/snapshots/153/fs
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr varIn overlayfs case, the equivalent layer only contains the opt/file hardlink:
$ ls /home/doc/.local/share/buildkit/runc-overlayfs/snapshots/snapshots/43/fs/
optSo this is expected to have all files from previous layers in the merged layer in fuse-overlayfs case ?
tonistiigi
commented
1 year ago There is a special optimization for certain snapshotters in https://github.com/moby/buildkit/blob/master/snapshot/merge.go#L19 that can do hardlinks. I'm not instantly sure if for fuse-overlayfs, there is a reason why the hardlink optimization is not possible at all or if it is just that it is not enabled(it might also need more fuse-specific work).
sipsma
commented
1 year ago There is a special optimization for certain snapshotters in https://github.com/moby/buildkit/blob/master/snapshot/merge.go#L19 that can do hardlinks. I'm not instantly sure if for fuse-overlayfs, there is a reason why the hardlink optimization is not possible at all or if it is just that it is not enabled(it might also need more fuse-specific work).
In theory it may be relatively straightforward to support fuse-overlayfs too; the hardlinking optimizations rely on the ability to get the underlying lowerdirs of the mount so they can be used to make hardlinks from the underlying filesystem, e.g. here: https://github.com/moby/buildkit/blob/2816a8328fb13b81bc26b6fd70c67be1ca86e5c6/snapshot/diffapply_unix.go#L171-L178
Provided the fuse-overlayfs snapshotter makes it easy to obtain those too (i.e. it doesn't just give you a bind mount sourced from a fuse mount or something indirect like that) then I think adding hardlink support for it would be fairly easy if anyone wants to contribute it.
Also, if support for this was added it would probably also make it trivial to support fuse-overlayfs in the overlay-optimized differ, which may provide other significant performance benefits.
amurzeau
commented
1 year ago I've tried a quick and dirty change:
"overlay" (like the switch you quoted) to "overlay || fuse3.*" (Type starts with "fuse3." with fuse-overlayfs)
setRedirectDir, I don't think fuse-overlayfs supports redirect_dir flag.hardlinkMergeSnapshotters and overlayBasedSnapshotters in merge.goAnd successfully run a build that used less disk space ! (the same disk usage as with overlayfs)
I've not checked everything yet, it seems too simple to be true ! There are probably edge cases to check.
amurzeau
commented
1 year ago I get Server message: toomanyrequests error when trying to execute integration tests:
httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/amd64/busybox/manifests/sha256:0d5a701f0ca53f38723108687add000e1922f812d4187dea7feaee85d2f5a6c5: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limitIs there a way to cache downloaded images or to pass login information ? I've tried to put my docker login credentials in ~/.docker/config.json within the dev-env container, but tests don't seem to read that file, the error still occurs.
I'm executing them in debug mode from VS Code in a devcontainer based on dev-env target from ./Dockerfile.
Hi,
Using fuse-overlayfs takes more disk space than overlayfs with COPY --link.
With fuse-overlayfs, the cache contains for the COPY --link:
One snapshot with COPY content Size: 1.05GB Description: copy /opt /opt/
One snapshot with COPY content + parent layer: Size: 1.16GB Description: [stage-1 2/3] LINK COPY --link --from=install /opt /opt/
With overlayfs, the cache contains:
One snapshot with COPY content Size: 1.05GB Description: copy /opt /opt/
One snapshot with COPY content hardlinked to another snapshot, thus taking no disk space: Size: 8.19kB Description: [stage-1 2/3] LINK COPY --link --from=install /opt /opt/
When doing COPY --link with a big folder, this more than double the cache disk usage for that COPY --link operation with fuse-overlayfs. I there a way to avoid that ?
Dockerfile:
buildctl du --verbose output with overlayfs snapshotter:
buildctl du --verbose output with fuse-overlayfs snapshotter:
du inside buildkit folders
Versions:
I'm interested in fuse-overlayfs primarily to be able to use fuse-overlayfs on RHEL which doesn't allow overlayfs in rootless mode.