[RFE] Add support for `--link` in `COPY/ADD`

[lucacome]

Description

Buildkit added support for the --link flag. Quoting from the docs:

Enabling this flag in COPY or ADD commands allows you to copy files with enhanced semantics where your files remain independent on their own layer and don't get invalidated when commands on previous layers are changed.

When --link is used your source files are copied into an empty destination directory. That directory is turned into a layer that is linked on top of your previous state.

This is a feature request to add this feature in buildah 🙂

[CypherpunkSamurai]

Hello @lucacome and @TomSweeneyRedHat,

I want to contribute to this issue but am new to the project, How can I get started? :)

[rhatdan]

Well first you would need to add support for --link in https://github.com/openshift/imagebuilder I believe. @flouthoc Could you help @CypherpunkSamurai contribute.

[flouthoc]

Hi,

Yes implementation must start from imagebuilder and introduce a new attribute link bool to Copy struct and must be set true when COPY or ADD contains --link and on buildah side if Copy.link == truethen layer must be created on a scratch base, in theory COPY --link must be replaced with a functionality similar to stage where scratch image is base and file is just copied to target path.

COPY --link /src /target

must create a intermediate image equivalent to

FROM scatch
COPY /src /target

[lucacome]

Hi @CypherpunkSamurai

I was wondering if you've had a chance to start working on this? Or do you need any help? 🙂

[CypherpunkSamurai]

Really sorry about this, I had gotten a little busy with college work.

I'll start work asap by this Saturday.

[rhatdan]

No problem college work should come first.

[CypherpunkSamurai]

Hello everyone, really sorry again for stalling this issue.

Ok Here's what I understand from the documentation of the --link tag for COPY and ADD build commands:

• It creates a cached layer that can be reused
• This cached layer is a layer from scratch. (i.e. a simple rootfs with busybox ? doubts here)

Here's how I understand how I need to implement it:

• introduce a new attribute link bool to Copy struct and must be set true when COPY or ADD contains --link
• on buildah side if Copy.link == truethen layer must be created on a scratch base, in theory COPY --link must be replaced with a functionality similar to stage where scratch image is base and file is just copied to target path.

Please check if this is correct

[flouthoc]

Yes SGTM. Pretty much what's written here https://github.com/containers/buildah/issues/4325#issuecomment-1286568410 but I'd suggest if you have time to take a look and play with buildkit a bit before implementing this. My explanation in above comment is more conceptual and in past we have seen that buildkit's feature is not completely described in the documentation so its worth verifying edge cases before implementing this.

Note: Buildah mostly tries to match parity with buildkit as much as possible.

[CypherpunkSamurai]

Sure. I need to try both the tools first i guess. Would be interesting. I'll try my best to help this issue :)

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[CypherpunkSamurai]

Alright I researched quite a bit about rootfs and build root for the few weeks. I feel comfortable with linux rootfs and how it works.

Um, if you don't mind guys (@flouthoc and @rhatdan) can you please help me point out how buildah handles creating scratch rootfs? (in addition line number would be very cool)

[flouthoc]

@CypherpunkSamurai Its hard to pin the exact line since I'll have to skim the code but I can tell that builder creates a container for FROM scratch instruction you can actually find the part where builder actually creates container.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since we have not heard any more feedback, closing. Reopen if you want to continue looking into this one.

[lucacome]

@rhatdan I'm still interested in this and I'd like to keep it open. Maybe somebody else can take over from @CypherpunkSamurai ?

[flouthoc]

@CypherpunkSamurai If you are running short on time, I'd request to assign this issue to me :)

[CypherpunkSamurai]

Sure. I'm having my exams right now, probably not a great time to work on this. I better not keep this issue stalling :)

On 24 January 2023 10:18:08 AM flouthoc @.***> wrote:

@CypherpunkSamurai If you are running short on time, I'd request to assign this issue to me :) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[der-eismann]

Hey @flouthoc, any updates on this? This feature would be great to have :slightly_smiling_face:

[veritas9872]

I am also very interested.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[sanmai-NL]

Not stale.

[lucacome]

Any updates on this?

buildkit states concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit so I was also wondering why podman is not using buildkit instead of buildah.

It seems like a lot of work to try and keep parity with buildkit and that effort could be put into improving buildkit? 🤷‍♂️

[rhatdan]

@flouthoc This is rising in importance since Podman-desktop needs this.

[BlaineEXE]

Ditto, want this functionality

[johnlauder]

Bump, would like this functionality. Currently have to modify Dockerfiles to work with podman due to this issue.

[sastorsl]

@CypherpunkSamurai did you have any work done on this you could share, even if incomplete? I could try and contribute, and if I'm not able - some ground work could be of guidance.

[sigbits]

I gave it a try and switched from docker to podman, but without COPY --link it's useless for my projects. Since the issue is already pretty old: is there any chance this will be fixed any time in the near future?

[huww98]

I’m currently running buildkitd in podman, then use buildctl to build images.

[danishprakash]

@flouthoc If you haven't started, I can take a shot at this.

[rhatdan]

@danishprakash go for it. @flouthoc has gone back to school and has much less time to work on this.

[edwbuck]

Any updates on this one?

[apollo13]

I'd love to see this as well. But I am kinda wondering if it makes sense to reimplement what buildkit does instead of using/embedding buildkit in podman? FWIW, aside from being daemonless (though you kind of can run buildkit like that as well: https://github.com/moby/buildkit?tab=readme-ov-file#daemonless ) podman build imo no longer has any features over docker build(x). The performance gains with buildkits are so big that I am currently reworking a few of our internal pipelines to use buildkit.

[flouthoc]

But I am kinda wondering if it makes sense to reimplement what buildkit does instead of using/embedding buildkit in podman?

@apollo13 Could you please elaborate on this, podman does not uses or embed buildkit as-is most of the features are implemented in buildah as per buildah's design and podman uses buildah not buildkit.

[apollo13]

@flouthoc I was mainly wondering if it makes sense (or were possible) for podman to directly use buildkit instead of buildah. This would allow keeping feature parity more easily and also pick up all performance improvements. Especially with multi stage builds podman is really really much slower since buildkit is able to parallelize many things (unless I missed a flag for podman build)

[flouthoc]

Podman multi-stage build is parallel but you have to use --jobs flag. See https://docs.podman.io/en/latest/markdown/podman-build.1.html#jobs-number

[apollo13]

Oh thank you, how did I miss that (is it new?). That does indeed help for multi-stage builds. Can you recommend any other knobs to tune?

On Sun, Aug 18, 2024, at 22:17, flouthoc wrote:

Podman multi-stage build is parallel but you have to use --jobs flag. See https://docs.podman.io/en/latest/markdown/podman-build.1.html#jobs-number

— Reply to this email directly, view it on GitHub https://github.com/containers/buildah/issues/4325#issuecomment-2295378163, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAT5C7QBYVLOL3FLASV2D3ZSD6OBAVCNFSM6AAAAAAQ72EQZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJVGM3TQMJWGM. You are receiving this because you were mentioned.Message ID: @.***>