earthly / earthly

Super simple build framework with fast, repeatable builds and an instantly familiar syntax – like Dockerfile and Makefile had a baby.
https://earthly.dev
Mozilla Public License 2.0
11.36k stars 397 forks source link

Can I `COPY` artifacts from a matrix build? #4161

Open negz opened 4 months ago

negz commented 4 months ago

Question

I'd like to compile Go binaries for a bunch of different platforms and upload the resulting binaries to S3, but I'm not sure how to do that (or if it's possible). The problem is that I'm not sure how to COPY an artifact from multiple platforms.

Here's a "pseudo Earthfile" showing roughly what I want to do:

build:
  ARG NATIVEPLATFORM
  ARG TARGETARCH
  ARG TARGETOS
  ARG GOARCH=${TARGETARCH}
  ARG GOOS=${TARGETOS}
  FROM --platform=${NATIVEPLATFORM} golang:1
  COPY --dir code .
  RUN go build -o binary ./code
  SAVE ARTIFACT binary

upload-binaries:
  FROM amazon/aws-cli
  COPY (+build/binary --platform linux/arm64 --platform linux/amd64) out/binary
  RUN --aws --push aws s3 sync out s3://some.bucket

This seems to result in strange behaviour. Specifically I only see the build target run for amd64, but the resulting binary appears to be arm64.

That said, even if it did compile the binary for both platforms, how would I specify using the COPY syntax that I'd like the different binaries for different platforms copied to different paths?

Ideally I could write something like:

  COPY (+build/binary --platform linux/arm64 --platform linux/amd64) out/${PLATFORM}/binary
negz commented 4 months ago

FWIW I'm currently just working around this by having some target SAVE ARTIFACT binary AS LOCAL out/${GOOS}_${GOARCH}/binary, then having another target COPY and push the out directory. I know that's considered an anti-pattern though.

alexcb commented 4 months ago

I'm not even sure if it works with args (rather than platform), e.g.

VERSION 0.8

echo:
  FROM alpine
  ARG foo
  ARG bar
  RUN echo $foo > "file.$bar"
  SAVE ARTIFACT file*

test-matrix:
  FROM alpine
  WORKDIR /files
  COPY (+echo/file* --foo=a --bar=b --bar=c) .
  RUN ls -la

produces:

        +test-matrix | --> FROM +base
              alpine | --> Load metadata alpine linux/amd64
               +echo | bar=c foo=a
               +echo | --> FROM +base
               +echo | bar=c foo=a
               +echo | --> FROM alpine
               +echo | [----------] 100% FROM alpine
               +echo | bar=c foo=a
               +echo | --> RUN echo $foo > "file.$bar"
        +test-matrix | --> WORKDIR /files
        +test-matrix | --> COPY (foo=a bar=b bar=c) +echo/file* ./
        +test-matrix | --> RUN ls -la
        +test-matrix | total 12
        +test-matrix | drwxr-xr-x    1 root     root          4096 May 27 16:29 .
        +test-matrix | drwxr-xr-x    1 root     root          4096 May 27 16:29 ..
        +test-matrix | -rw-r--r--    1 root     root             2 Apr 16  2020 file.c