BUG: global ARG not shown as build step

[h-vetinari]

[Continuing from #1456; containers/libpod#2250]

This is a comparatively small bug, but confusing nevertheless. In multistage builds, it may be necessary to re-use a build-arg for different stages (I'm using the generic DEBIAN_FRONTEND in the example below), but setting it per stage would lead to an error: ARG requires exactly one argument definition

Docker solves this by being able to specify a global ARG before the first FROM, cf. here. Podman/buildah execute this correctly, but does not show the ARG before the first FROM as a build step.

Say we have the following dockerfile, and call it podman_test.dockrf:

ARG DEBIAN_FRONTEND=noninteractive

FROM docker.io/library/ubuntu:bionic as base
ARG DEBIAN_FRONTEND
RUN echo $DEBIAN_FRONTEND

RUN apt-get update && apt-get install curl -y --no-install-recommends

FROM base as main
ARG DEBIAN_FRONTEND
RUN echo $DEBIAN_FRONTEND

RUN apt-get update && apt-get install wget -y --no-install-recommends

CMD ["bash"]

Then the build log is

$ sudo podman build -t podman_test -f podman_test.dockrf .
STEP 1: FROM docker.io/library/ubuntu:bionic AS base
STEP 2: ARG DEBIAN_FRONTEND
--> 8dbc2806a4f15de339706b0512a21e8a7453da79a07101900b0afe077e029c10
STEP 3: FROM 8dbc2806a4f15de339706b0512a21e8a7453da79a07101900b0afe077e029c10 AS base
STEP 4: RUN echo $DEBIAN_FRONTEND
noninteractive
--> 4c48e1473c102e397ff4381bae5bfdba811313894c1c159a525e19975e0eb9a0
STEP 5: FROM 4c48e1473c102e397ff4381bae5bfdba811313894c1c159a525e19975e0eb9a0 AS base
STEP 6: RUN apt-get update && apt-get install curl -y --no-install-recommends
[snip]
--> abf1a7cf75a5da41939ebb1ac25e74731ea9ce3cf81815466bdc960bd1fa6be8
STEP 7: COMMIT podman_test
--> abf1a7cf75a5da41939ebb1ac25e74731ea9ce3cf81815466bdc960bd1fa6be8
--> 8b188946f04cd8c364c9534e5b565718956dcfa98d6c9f25924a4089a4f7296e
STEP 8: FROM base AS main
STEP 9: ARG DEBIAN_FRONTEND
--> 63454d79b0cea53dd6b6e23a012526d6b3cb7b1669028d64a1677601520ab649
STEP 10: FROM 63454d79b0cea53dd6b6e23a012526d6b3cb7b1669028d64a1677601520ab649 AS main
STEP 11: RUN echo $DEBIAN_FRONTEND
noninteractive
--> eb6f70088197b1e2f9ad95a52d82f68e208642def1903785d1dd909ccfa3c2f7
STEP 12: FROM eb6f70088197b1e2f9ad95a52d82f68e208642def1903785d1dd909ccfa3c2f7 AS main
STEP 13: RUN apt-get update && apt-get install wget -y --no-install-recommends
[snip]
--> bd5ec6d851d72a0aee436fc3683efd00d6cbd8bfa58d97047787888ccee283fd
STEP 14: FROM bd5ec6d851d72a0aee436fc3683efd00d6cbd8bfa58d97047787888ccee283fd AS main
STEP 15: CMD ["bash"]
--> 93b7c31fab24cfb97f8b8b3d158beaeb59710fff81c1c9730cf417e3b39c5f52
STEP 16: COMMIT podman_test
--> 93b7c31fab24cfb97f8b8b3d158beaeb59710fff81c1c9730cf417e3b39c5f52
--> 083b41e0fee06ff7ef9e0a182d872f19e29ae53886daff2283cb317edc95d2d4

where one can see that the variable is inserted into the stages correctly (cf. RUN echo $DEBIAN_FRONTEND), but the first step is not shown (as docker does).

[rhatdan]

Looks like this has been fixed.

[h-vetinari]

@rhatdan Just checked this with the latest Ubuntu PPA, and it has not been fixed - please reopen.

The issue is the missing line for the first ARG in the build log (and correspondingly different number of build steps):

Compare:

$ sudo podman build -t podman_test -f podman_test.dockrf .
STEP 1: FROM docker.io/library/ubuntu:bionic AS base
STEP 2: ARG DEBIAN_FRONTEND
--> f7a3d1782ddcd19b51f0645d2f26db69c59f343857064a10d6d3933ac0474d27
STEP 3: RUN echo $DEBIAN_FRONTEND
noninteractive
--> bfa0f8aa35d45432be72449729c0a99a292fe5f8e4aa150181e7a33e626e5446
STEP 4: RUN apt-get update && apt-get install curl -y --no-install-recommends

with:

$ sudo docker build -t podman_test -f podman_test.dockrf .
Sending build context to Docker daemon   68.1kB
Step 1/10 : ARG DEBIAN_FRONTEND=noninteractive
Step 2/10 : FROM docker.io/library/ubuntu:bionic as base
 ---> 7698f282e524
Step 3/10 : ARG DEBIAN_FRONTEND
 ---> Running in a0ef7256a8ec
Removing intermediate container a0ef7256a8ec
 ---> 04d8681b1c2d
Step 4/10 : RUN echo $DEBIAN_FRONTEND
 ---> Running in 1331c384f587
noninteractive
Removing intermediate container 1331c384f587
 ---> 148d3df9424a
Step 5/10 : RUN apt-get update && apt-get install curl -y --no-install-recommends

[rhatdan]

@ashley-cui PTAL

[rhatdan]

@TomSweeneyRedHat Could you confirm if this is still a bug in v1.9?

[h-vetinari]

@rhatdan

I checked with the newest ubuntu ppa (which already has 1.9.0), and it is still an issue.

[TomSweeneyRedHat]

Still an issue in 1.15.0-dev

[carbonin]

This is actually an artifact of how openshift/imagebuilder handles "heading" args.

Specifically extractHeadingArgsFromNode removes the initial ARG commands from the tree so they are never seen by buildah.

The resulting image should be correct, but you're right in that buildah wouldn't print that first line. We could put in a special case and no-op any ARG commands before the first from just so we could print the step, but that feels like it's bound to break as we really don't want that ARG processed as if it was a part of the first stage.

I don't have a great solution, but I'm wondering if this is really something that needs to be fixed given that the image is correct.

[h-vetinari]

It's clearly not a critical issue, but I think ARG-before-FROM should appear as a build step, otherwise it breaks both the principle of least surprise, as well as the parity to docker.

[rhatdan]

Could one of you two @carbonin @h-vetinari open a PR to fix this issue?

[rhatdan]

Well no one has picked this up and it is not critical, so I will close if no one in community grabs it in the next month,

[h-vetinari]

Hm, that's IMO not a good reason to close . Yes, criticality is low, but that doesn't mean it shouldn't be fixed eventually. I haven't looked at the buildah-code, but @carbonin's link is to openshift/imagebuilder - is that really the place to change it?

[rhatdan]

Most likely yes. So we could open an issue over there. I am not a huge fan of just leaving issues out wasting away on the vine, in the hopes that someone will eventually fix it. But if a fix went into imagebuilder, buildah would use it.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[smac89]

I just noticed this today as well. It is a bit of a hindrance to multistage builds, and contrary to the docker specification on how ARG is treated.

Here is a simple test:

buildah bud --no-cache --build-arg foo=beat -<<'EOF' 
ARG foo

FROM alpine:latest as base
ENV foo="${foo:-booze}"
ENV bar="feel the"

FROM base as stage0
RUN echo "$bar" && echo "$foo"

FROM alpine:latest as stage1
RUN echo "$bar" && echo "$foo"
EOF

Outputs:

STEP 1: FROM alpine:latest AS base
STEP 2: ENV foo="${foo:-booze}"
STEP 3: ENV bar="feel the"
--> 18723566166
STEP 4: FROM alpine:latest AS stage1
STEP 5: RUN echo "$bar" && echo "$foo"

STEP 6: COMMIT
--> 18d1aa0e367
STEP 7: FROM 187235661667502aabd8d75077d62caed38c288a2c2f457ce22b636496fc62d4 AS stage0
STEP 8: RUN echo "$bar" && echo "$foo"
feel the
booze
18d1aa0e36756d10fd9da3f736ee64b81c74c314536ff22fd6d1ebf9c1f5cfec

The output from step 8 should have been beat, not booze, if this was working correctly. Please consider making this a priority because multistage builds which do not reuse previous images as new base, will be impossible to acheive if they require passing args to later stages.

Thanks

[carbonin]

This issue is specifically about the output from the buildah command as compared to docker. Both tools currently treat multi-stage builds the same for me.

Running a docker and buildah build with your Dockerfile proves this out:

$ cat Dockerfile 
ARG foo

FROM alpine:latest as base
ENV foo="${foo:-booze}"
ENV bar="feel the"

FROM base as stage0
RUN echo "$bar" && echo "$foo"

FROM alpine:latest as stage1
RUN echo "$bar" && echo "$foo"

$ docker build --no-cache --build-arg foo=beat .
Sending build context to Docker daemon  2.048kB
Step 1/8 : ARG foo
Step 2/8 : FROM alpine:latest as base
 ---> 6dbb9cc54074
Step 3/8 : ENV foo="${foo:-booze}"
 ---> Running in 836e00d5096f
Removing intermediate container 836e00d5096f
 ---> 8771332970bb
Step 4/8 : ENV bar="feel the"
 ---> Running in 6378fa42ba43
Removing intermediate container 6378fa42ba43
 ---> d7bd41f72074
Step 5/8 : FROM base as stage0
 ---> d7bd41f72074
Step 6/8 : RUN echo "$bar" && echo "$foo"
 ---> Running in a1737d4768c7
feel the
booze
Removing intermediate container a1737d4768c7
 ---> f68e316512b6
Step 7/8 : FROM alpine:latest as stage1
 ---> 6dbb9cc54074
Step 8/8 : RUN echo "$bar" && echo "$foo"
 ---> Running in 53e62d34885d

Removing intermediate container 53e62d34885d
 ---> c61bb6149de3
Successfully built c61bb6149de3

$ buildah bud --no-cache --build-arg foo=beat .
STEP 1: FROM alpine:latest AS base
STEP 2: ENV foo="${foo:-booze}"
STEP 3: ENV bar="feel the"
--> 4f2be3f0050
STEP 4: FROM 4f2be3f0050e2a58e445c435f22fd49d73cb89b0a15e9db7fdf2dda4be710a80 AS stage0
STEP 5: RUN echo "$bar" && echo "$foo"
feel the
booze
STEP 6: FROM alpine:latest AS stage1
STEP 7: RUN echo "$bar" && echo "$foo"

STEP 8: COMMIT
--> 706a5996d41
706a5996d419c3ede3ea4144806ef724eba1c8b667e5caeafcdda50699b3761e

$ buildah version
Version:         1.18.0
Go Version:      go1.14.10
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.8.0
Git Commit:      
Built:           Wed Dec 31 19:00:00 1969
OS/Arch:         linux/amd64

$ docker version
Client: Docker Engine - Community
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c289
 Built:             Fri Apr  9 22:47:32 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:45:12 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I also included my buildah and docker versions for reference. Maybe this regressed? If so, please open a new issue, as, like I said, this one is specifically about how the steps are printed.

[rhatdan]

@TomSweeneyRedHat Would you take a look at this, you have more experience in this area, then me.

[rhatdan]

I am seeing feel the booze from Docker was well

# docker build --no-cache --build-arg foo=beat -<<'EOF' 
ARG foo

FROM alpine:latest as base
ENV foo="${foo:-booze}"
ENV bar="feel the"

FROM base as stage0
RUN echo "$bar" && echo "$foo"

FROM alpine:latest as stage1
RUN echo "$bar" && echo "$foo"
EOF
Sending build context to Docker daemon  2.048kB
Step 1/8 : ARG foo
Step 2/8 : FROM alpine:latest as base
 ---> 28f6e2705743
Step 3/8 : ENV foo="${foo:-booze}"
 ---> Running in 51a3673b6622
Removing intermediate container 51a3673b6622
 ---> f3fa5543169a
Step 4/8 : ENV bar="feel the"
 ---> Running in 5984816bcdd7
Removing intermediate container 5984816bcdd7
 ---> 6fdfbc3ec9d8
Step 5/8 : FROM base as stage0
 ---> 6fdfbc3ec9d8
Step 6/8 : RUN echo "$bar" && echo "$foo"
 ---> Running in 4331f85db79e
feel the
booze
Removing intermediate container 4331f85db79e
 ---> ed5b22041136
Step 7/8 : FROM alpine:latest as stage1
 ---> 28f6e2705743
Step 8/8 : RUN echo "$bar" && echo "$foo"
 ---> Running in 58dd685e87b3

Removing intermediate container 58dd685e87b3
 ---> 80d2f4407a29
Successfully built 80d2f4407a29
sh-5.1# 

If I move the ARG to after the from, everything works.

docker build --no-cache --build-arg foo=beat -<<'EOF' 
FROM alpine:latest as base
ARG foo
ENV foo="${foo:-booze}"
ENV bar="feel the"

FROM base as stage0
RUN echo "$bar" && echo "$foo"

FROM alpine:latest as stage1
RUN echo "$bar" && echo "$foo"
EOF
Sending build context to Docker daemon  2.048kB
Step 1/8 : FROM alpine:latest as base
 ---> 28f6e2705743
Step 2/8 : ARG foo
 ---> Running in 7f8f41c8fdbe
Removing intermediate container 7f8f41c8fdbe
 ---> c6a164e3893d
Step 3/8 : ENV foo="${foo:-booze}"
 ---> Running in 5af12185c0c5
Removing intermediate container 5af12185c0c5
 ---> 2d206c907146
Step 4/8 : ENV bar="feel the "
 ---> Running in 9ea37430b0f3
Removing intermediate container 9ea37430b0f3
 ---> d98f2e9a24dc
Step 5/8 : FROM base as stage0
 ---> d98f2e9a24dc
Step 6/8 : RUN echo -n "$bar" && echo "$foo"
 ---> Running in fa0bdddb0bd5
feel the beat
Removing intermediate container fa0bdddb0bd5
 ---> 270849e5c2b6
Step 7/8 : FROM alpine:latest as stage1
 ---> 28f6e2705743
Step 8/8 : RUN echo "$bar" && echo "$foo"
 ---> Running in 3368137bad99
Removing intermediate container 3368137bad99
 ---> 4312cc8bb3c3
Successfully built 4312cc8bb3c3

 buildah bud --no-cache --build-arg foo=beat -<<'EOF' 

FROM alpine:latest as base
ARG foo
ENV foo="${foo:-booze}"
ENV bar="feel the"

FROM base as stage0
RUN echo -n "$bar" && echo "$foo"

FROM alpine:latest as stage1
RUN echo "$bar" && echo "$foo"
EOF
STEP 1: FROM alpine:latest AS base
STEP 2: ARG foo
STEP 3: ENV foo="${foo:-booze}"
STEP 4: ENV bar="feel the "
--> 3265baf9274
STEP 5: FROM 3265baf9274e57b4753631206d22872ca0880c1b03a6de04857d327e53888a19 AS stage0
STEP 6: RUN echo -n "$bar" && echo "$foo"
feel the beat
STEP 7: FROM alpine:latest AS stage1
STEP 8: RUN echo "$bar" && echo "$foo"

STEP 9: COMMIT
--> 528b37ee993
528b37ee9939f72d624faa5563890b788f18878304c857c4a1c15f2dd1fdc0f6

[TomSweeneyRedHat]

Finally getting to this, but yes, @rhatdan is correct. The 'FROM' wipes out previous values in ARG.