Cryptic Message when Starting Earthly: 403 Forbidden

[BenAlanSouthall]

I have a relatively straightforward Earthfile, and am using Earthly 0.6.27. I would need to redact in order to share here, but I'm not sure if it's relevant:

Running earthly +build spits out the following:

 2. Build 🔧
————————————————————————————————————————————————————————————————————————————————

r/s/g/foo:focal | --> Load metadata linux/amd64
r/s/g/foo:focal | WARN: The command 'Load metadata linux/amd64' failed: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
Share your logs with an Earthly account (experimental)! Register for one at https://ci.earthly.dev.
Error: build target: build main: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden

This was working previously, but I am now running as a different user.

I'm running Earthly with rootless podman - I know this technically isn't supported yet, but so far I ahve not had any major problems.

I do regularly get ifferent messages about not being able to pull the FROM image - when I hit these, I run podman pull manually and retry a few times, until it goes. This seems to fail very quickly however. It's not really clear to me what is failing either.

Thanks!

[vladaionescu]

The image name you're trying to use seems incorrect? Is it really called foo?

(r/s/g/foo:focal)

[BenAlanSouthall]

No, I've simply redacted the real basename as it's confidential ;)

[BenAlanSouthall]

I found the -V switch:

 1. Init 🚀
————————————————————————————————————————————————————————————————————————————————

            frontend | Running command: podman info --format={{.Host.Security.Rootless}}
            frontend | podman-shell frontend initialized.
            frontend | Running command: podman container inspect earthly-buildkitd
           buildkitd | Found buildkit daemon as podman container (earthly-buildkitd)
            frontend | Running command: podman container inspect earthly-buildkitd
            frontend | Running command: podman image inspect earthly/buildkitd:v0.6.27
           buildkitd | Comparing running container image ("4e0a15f3253a232b2683f183f954b06ecfdbfad7bcfafcfcaa3dee728e2fa79d") with available image ("4e0a15f3253a232b2683f183f954b06ecfdbfad7bcfafcfcaa3dee728e2fa79d")
            frontend | Running command: podman container inspect earthly-buildkitd
           buildkitd | Settings hashes match ("f058b40f9ff49be6"), no restart required
           buildkitd | Version github.com/earthly/buildkit v0.6.27 9c256c19c5397aee1ba0379590d685c2e87ff17c
           buildkitd | Platforms: linux/amd64 (native) linux/amd64/v2 linux/amd64/v3 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/arm/v7 linux/arm/v6
           buildkitd | Utilization: 0 other builds, 0/20 op load
           buildkitd | GC stats: 0 B cache, avg GC duration 0s, all-time GC duration 0s, last GC duration 0s, last cleared 0 B

 2. Build 🔧
————————————————————————————————————————————————————————————————————————————————

r/s/g/foo:focal | --> Load metadata linux/amd64
r/s/g/foo:focal | WARN: The command 'Load metadata linux/amd64' failed: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
r/s/g/foo:focal | Overriding args used: 
r/s/g/foo:focal | Completed in 389.87249ms
earthfile2llb immediate error: Earthfile line 29:0 apply FROM: apply build +base: earthfile2llb for +base: Earthfile line 4:0 apply FROM registry.gitlab.com/some-project/general-registry/foo:focal: resolve image config for registry.gitlab.com/some-project/general-registry/foo:focal: failed to fetch anonymous token: unexpected status: 403 Forbidden
in              +base
in              +install
earthfile2llb group error: <nil>
Summary of timing information
Note that the times do not include the expansion of commands like BUILD, FROM, COPY (artifact).
r/s/g/foo:focal | () 389.87249ms
===============================================================
Total           389.87249ms
Total (real)    433.430659ms
Share your logs with an Earthly account (experimental)! Register for one at https://ci.earthly.dev.
Error stack trace:
github.com/moby/buildkit/util/stack.Enable
        /go/pkg/mod/github.com/earthly/buildkit@v0.0.1-0.20220913002628-9c256c19c539/util/stack/stack.go:77
github.com/moby/buildkit/util/grpcerrors.FromGRPC
        /go/pkg/mod/github.com/earthly/buildkit@v0.0.1-0.20220913002628-9c256c19c539/util/grpcerrors/grpcerrors.go:188
github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
        /go/pkg/mod/github.com/earthly/buildkit@v0.0.1-0.20220913002628-9c256c19c539/util/grpcerrors/intercept.go:41
google.golang.org/grpc.(*ClientConn).Invoke
        /go/pkg/mod/google.golang.org/grpc@v1.47.0/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
        /go/pkg/mod/github.com/earthly/buildkit@v0.0.1-0.20220913002628-9c256c19c539/api/services/control/control.pb.go:1739
github.com/moby/buildkit/client.(*Client).solve.func2
        /go/pkg/mod/github.com/earthly/buildkit@v0.0.1-0.20220913002628-9c256c19c539/client/solve.go:244
golang.org/x/sync/errgroup.(*Group).Go.func1
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1581
Error: build target: build main: failed to solve: failed to fetch anonymous token: unexpected status: 403 Forbidden
Warning: timedout while sending analytics

The image is being held in gitlab upstream; but as mentioned, it's pulling okay for me from the podman CLI.

what is confusing is that I have managed to build so far as a different user. For this user, it's not happy.

[r-LaForge]

Hey @BenAlanSouthall

It may be the case that your environment variables between users differ. We use a custom Podman attachable when providing the necessary credentials back to buildkit.

As you can tell it relies on the XDG_RUNTIME_DIR environment variable being set to find the Podman auth and then falls back to trying to retrieve the credentials the "Docker" way.

Try setting this environment variable to the top-level folder where your config is held (by default it could be ~/.config).

In the meantime, I think earthly could benefit from a patch that has us fall back to that default folder if the environment variable isn't set.

[BenAlanSouthall]

Thanks for getting back to me!

Ah! That makes sense, as this machine has a graphical login - the difference being that when running Earthly as myself, I have already got a seat and so this has set up the XDG_RUNTIME dirs; for any of my colleagues' accounts, this won't be the case as between all of us we will only accessing via sudo or from SSH.

1) Is there a way I can get the logging at that level so that messages lik that would be printed in future? 2) I will need to refresh myself on the best way for ensuring that a session is created appropriately for console-/SSH-based logins, and then once I have this I will confirm that it's no longer reproducible 3) Falling back sounds good to me, and I think still the original message isn't the most clear as to what internal component is failing, so in terms of usefully contributing changes from this bug, I'd probably suggest ideally the message that comes out in this situation could be improved.

Thanks for your help!

[r-LaForge]

Just a follow-up that we have a proposed change here that we still need to test before pulling in: https://github.com/earthly/buildkit/pull/105/files

Is there a way I can get the logging at that level so that messages like that would be printed in future? Yes, I think ideally we could at least print a warning indicating we couldn't find the configuration file or that the environment variable was not set.

Let me know if you've been able to confirm this is no longer an issue after setting that variable and ensuring the configuration is setup.