Closed ElSamhaa closed 9 months ago
A friendly reminder that this issue had no activity for 30 days.
I tried this with Docker, and it fails?
# systemctl start docker
# cat /tmp/test/Dockerfile
FROM alpine
COPY <<eof /tmp/hello
hello there
eof
# docker build /tmp/test/
Sending build context to Docker daemon 2.048kB
Error response from daemon: dockerfile parse error line 4: unknown instruction: HELLO
Ok But this still breaks on docker buildx build at least in the version of Docker i have installed.
# docker buildx build /tmp/test/
[+] Building 0.2s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 150B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 2B 0.0s
=> [1/2] FROM docker.io/library/alpine 0.0s
=> ERROR [2/2] COPY <<eof /tmp/hello hello there eof 0.0s
------
> [2/2] COPY <<eof /tmp/hello hello there eof:
------
error: failed to solve: failed to compute cache key: "/hello" not found: not found
# docker buildx build /tmp/test/
[+] Building 0.1s (2/2) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 89B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
error: failed to solve: failed to solve with frontend dockerfile.v0: failed to create LLB definition: dockerfile parse error line 4: unknown instruction: HELLO
My understanding is that this requires the correct # syntax=docker/dockerfile:1.3-labs
at the top of the Dockerfile.
Yup that works.
# docker buildx build /tmp/test/
[+] Building 5.9s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 184B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.3-labs 1.3s
=> docker-image://docker.io/docker/dockerfile:1.3-labs@sha256:03ca0e50aa4 4.3s
=> => resolve docker.io/docker/dockerfile:1.3-labs@sha256:03ca0e50aa4b6e7 0.0s
=> => sha256:ce897ecde42e8c45c0056748ec1a17a3edc5b7d3b091 9.67MB / 9.67MB 4.1s
=> => sha256:03ca0e50aa4b6e76365fa9a5607c3f988bc9284de6a8 2.00kB / 2.00kB 0.0s
=> => sha256:c6afe91d6f5e32ee187adf7516f1ced1d7d2f9621ff4e907 528B / 528B 0.0s
=> => sha256:456bbe8ddbe759d5109d4f5bfe8ce105c0086863298f 1.21kB / 1.21kB 0.0s
=> => extracting sha256:ce897ecde42e8c45c0056748ec1a17a3edc5b7d3b091dae35 0.1s
=> [internal] load .dockerignore 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> CACHED [1/2] FROM docker.io/library/alpine 0.0s
=> [internal] preparing inline document 0.0s
=> [2/2] COPY <<eof /tmp/hello 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:c14ea19390b6e4241141c42a5fb74abccf9c22af8f9a48 0.0s
Do you think we should support it?
Yeah, but I'm not working on the patches yet.
A friendly reminder that this issue had no activity for 30 days.
@flouthoc PTAL.
A friendly reminder that this issue had no activity for 30 days.
@flouthoc Did you ever get a chance to look at this?
I'll check this thanks.
A friendly reminder that this issue had no activity for 30 days.
@flouthoc any progress?
Ackd. Not yet, will start working on this.
It's a labs (experimental?) feature.
Meanwhile it's supported by default by BuildKit >= v0.10.1. (That BuildKit version was the first one to include a Dockerfile frontend as the built-in one with a syntax >= 1.4 (1.4.1), which is required for here-documents.)
Any ETA for when this might be resolved?
I am sure PRs would help move it along. :^) I don't believe anyone has looked at this yet.
Also impacted by this issue
please fix
Picking this up thanks, will report updates here.
I'm adding this here cause it wasn't clear which dockerfile works and which one doesn't , the syntax
header and DOCKER_BUILDKIT
are necessary for heredoc
blobs to work.
❯ cat Dockerfile
# syntax=docker/dockerfile:1.3-labs
from alpine
RUN <<EOF
uname -a
EOF
/tmp/d
❯ DOCKER_BUILDKIT=1 docker build .
[+] Building 1.3s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.3-labs 0.6s
=> CACHED docker-image://docker.io/docker/dockerfile:1.3-labs@sha256:250ce669e1aeeb5ffb892b18039c3f0801466536cb4210c8eb2638e628859bfd 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> [internal] load .dockerignore 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.5s
=> [1/2] FROM docker.io/library/alpine@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad 0.0s
=> CACHED [2/2] RUN <<EOF (uname -a) 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:9dec2cf343c991646e5b7dabec10f9b06bac59055b654b9b7454f363bc51bdab 0.0s
/tmp/d
❯ DOCKER_BUILDKIT=1 podman build .
STEP 1/4: FROM alpine
STEP 2/4: RUN <<EOF
error running container: error from /home/jgarcia/Projects/go/bin/crun creating container for [/bin/sh -c <<EOF]: 2022/10/19 09:12:22 Defaulting to port 8080
2022/10/19 09:12:22 Listening on port 8080
2022/10/19 09:12:22 listen tcp :8080: bind: address already in use
: exit status 1
Error: error building at STEP "RUN <<EOF": error while running runtime: exit status 1
I started work on this a while ago but got delayed due to other stuff, I think we can reuse parts from buildkit https://github.com/moby/buildkit/blob/c717d6aa7543d4b83395e0552ef2eb311f563aab/frontend/dockerfile/parser/parser.go#L477 and implement this. I'll get back to this soon.
Would you mind pointing me to the branch where you started the work ? i wouldn't mind taking a look and possibly help.
We can definitely port this to podmans parser , i was trying to get my head around it but this seems to be the critical bit: https://github.com/moby/buildkit/blob/master/frontend/dockerfile/parser/parser.go#L119
reHeredoc = regexp.MustCompile(`^(\d*)<<(-?)([^<]*)$`)
Im gonna try to investigate how this is evaluated later on
Would you mind pointing me to the branch where you started the work ? i wouldn't mind taking a look and possibly help.
I don't have any presentable branch yet but I think the work must
Start from imagebuilder (https://github.com/openshift/imagebuilder/) and we must populate heredoc parsed tree there and pass it to buildah.
On buildah side we have to dump the heredoc into a temporary file and mount it inside the builder container and execute it for the RUN
step after completion of RUN
step the temporary mounted heredoc file must be removed and unmounted.
I think If i'm able to open a draft PR we can discuss it better there.
100% that's what they seem to do on moby ,let me know if you open that PR .
A friendly reminder that this issue had no activity for 30 days.
@flouthoc Any movement on this one?
that would really be a cool feature to have
Waiting for this.
I already have somewhat working code for this I'll open a draft PR soon.
Great, and for those complaining and stating that they are waiting... The best way to get a feature into an opensource project is to roll up ones sleeves and create a PR.
+1 heredoc would be great.
If anyone lands here and finds themselves needing an alternate approach / workaround there are a few angles of attack. The following example is not as elegant as heredoc but I can confirm it's working with podman 4.3.1:
RUN echo '#!/usr/bin/env bash\n\
# example \n\
echo "hello world"' > /path/to/file.sh
Is this feature still not released? I tried to use this today and it was after reading an article from july 2021
Greetings! On the Docker website, it's now under Here-Documents / Creating inline files, and requires adding this line at the top of the Dockerfile
:
# syntax=docker/dockerfile:1
I just thought the syntax was cleaner than something like:
FROM alpine
RUN mkdir /app && echo -e "\
echo hello \${FOO}\n\
" > /app/script.sh
RUN FOO=abc ash /app/script.sh
@flouthoc @nalind lets talk about this next week.
Is this available via the REST API as well?
# syntax=docker/dockerfile:1.4-labs
FROM alpine
RUN <<EOF
apt-get update --fix-missing
EOF
This generates an error for apt-get Error: (HTTP code 400) unexpected - {"message":"dockerfile parse error line 10: unknown instruction: APT-GET"}
Note that this runs via. docker build .
I'm hoping that there is some movement on this soon. It would really be useful.
I should be ready with a PR soon, I have early infrastructure for this on the branches https://github.com/flouthoc/imagebuilder/commit/000f4710e1f0387a80d8a31760c9a20e29897a75 , https://github.com/flouthoc/imagebuilder/tree/heredoc and https://github.com/flouthoc/buildah/tree/heredoc-experiment
Okay my branch https://github.com/flouthoc/buildah/tree/heredoc-experiment and most recent commit i.e https://github.com/flouthoc/buildah/commit/270f3f9450eafc0a42cd7d83bcaf0359f5eec6ae has a working version with RUN
in case if someone wants to try it out.
A tryout Containerfile could be
FROM ubuntu:20.04
RUN <<EOF
echo "Hello" >> /hello
echo "World!" >> /hello
EOF
RUN ls -a
RUN cat hello
I will open a PR in coming days once i gets a ack on design from maintainers.
Thanks
@flouthoc maybe the best way to get the ack is to open the PR?
DOCKER_BUILDKIT=1 docker build . If you are running a version of Docker Engine version earlier than 23.0
@flouthoc THis is rising in importance since Podman-desktop needs this.
And FWIW, Mark Russel and I were talking about this yesterday. It would be good if we can squeeze this into 4.8.
This feature is merged into upstream via https://github.com/containers/buildah/pull/5092 and will be included in next release.
This is in Buildah v1.33.0 and above, FWIW
Description
Heredoc notation is supported in Dockerfile syntax, however buildah doesn't handle it correctly.
Steps to reproduce the issue: 1.Create a Dockerfile with the following content:
2.Run
podman build -t tst .
Describe the results you received:
Describe the results you expected: The image should build and have a file
/tmp/hello
that containhello world
.Output of
rpm -q buildah
orapt list buildah
:-
Output of
buildah version
:-
Output of
podman version
if reporting apodman build
issue:*Output of `cat /etc/release`:**
Output of
uname -a
:Output of
cat /etc/containers/storage.conf
: