Support running earthly rootless

[vladaionescu]

Currently Earthly starts a buildkit daemon in privileged mode. This is oftentimes incompatible with some CI environments (and also with k8s) which do not allow privileged operation.

An alternative to using the unprivileged environment of the CI the user can opt to connect to a remote buildkit instance. However, this requires that the user runs and operates such an instance on their own.

It would be useful to support rootless mode in Earthly (a feature that already exists in buildkit - it just needs to be ported). This should additionally make any WITH DOCKER instances used within builds also rootless.

[frafra]

Discussion on Slack: https://earthlycommunity.slack.com/archives/C01DL2928RM/p1648817327602409

Workaround

Earthfile:

#[...]
WORKDIR /app
#[...]
test-prepare:
    #[...]
    RUN --secret KEY1 --secret KEY2 echo "KEY1=${KEY1}\nKEY2=${KEY2}" > .env
    SAVE ARTIFACT .env .
    SAVE IMAGE myproject:test

test-local:
    LOCALLY
    COPY +test-prepare/.env .
    RUN docker-compose run --rm test ; docker-compose down -v ; rm .env

docker-compose.yml:

version: "3"
services:
  test:
    image: myproject:test
    command: pytest --cov=modules tests    
    env_file:
      - .env
    depends_on:
      - postgres
    volumes:
      - .:/app
  postgres:
    image: postgres:14
    environment:
      POSTGRES_HOST_AUTH_METHOD: trust

Notes

.env is necessary

Cannot pass secrets with LOCALLY: see #1800.

.env has to be removed at the end

I have no idea why, but this happens otherwise:

Error: secret "KEY1" already contains a value

Cannot have a single target

LOCALLY needs to be the first instruction, as it replaces FORM.

An artifact needs to be produced

It is not possible to use BUILD together with LOCALLY (as the documentation says), so even if .env is not needed, using COPY seems to be the only way to trigger the test-prepare stage implicitly.

[dchw]

It is not possible to use BUILD together with LOCALLY

This is a trick that feels similar to what I do when I need to control build order; and also closely approximates a BUILD:

#[...]
WORKDIR /app
#[...]
test-prepare:
    #[...]
    RUN --secret KEY1 --secret KEY2 echo "KEY1=${KEY1}\nKEY2=${KEY2}" > .env
    SAVE ARTIFACT .env AS LOCAL .env
    SAVE IMAGE myproject:test

test-local:
    LOCALLY
    RUN earthly +test-prepare
    RUN docker-compose run --rm test ; docker-compose down -v ; rm .env

This is fairly safe in a LOCALLY because of course earthly is installed! How else would we be running it? Its really close to how a BUILD works internally. This could let you separate the .env generation out into its own target at least, and sequester the workaround a little more.

[legend-zeratul]

Any luck on this issue? Has this been implemented?

Background: We're in the process of building a vertically integrated K8S platform around Gitops principles, and considering that Earthly streamlines almost all tasks we'll need to do, we're very much in favor of going ahead with Earthly.. Ideally we'd like it to be the same thing that does the build on the server as well.. So, build/preview locally using Earthly -> commit code -> something picks up the commit and calls Earthly on the server to build/publish the image -> CD picks up the image and updates the deployment.. The only only thing that is currently blocking us: we cannot have privileged mode on the server..

If we were to provide a rootless buildkit daemon that is also deployed on the cluster and point the server side Earthly-in-docker instance to this buildkit, will that work? (based on docs, our understanding is that the thing that needs to change is only the command used to start buildkit? or are we over simplifying what we will need to do?)

[legend-zeratul]

(happy to be corrected tbh - couldnt view the Slack thread on the topic, hence the qn..)

Could also help if someone can point out what we would lose if we had to run Earthly's buildkit in rootless mode..? Mounts? Ability to squash the layers in an image?

[frafra]

This has not been implemented and there is no timeline as far as I know. Earthly can be run on rootless Docker, but it is not able to run WITH DOCKER statements. No functionality would be lost by using rootless mode. It looks to me that the main issue is that Earthly tries to connect to the system socket instead of the user socket.

[legend-zeratul]

This has not been implemented and there is no timeline as far as I know. Earthly can be run on rootless Docker, but it is not able to run WITH DOCKER statements. No functionality would be lost by using rootless mode. It looks to me that the main issue is that Earthly tries to connect to the system socket instead of the user socket.

Hi, thanks for this - first glimmer of light.. Guess we can live with that for a while - not fully ideal (guess we cant run integration tests with compose on a cluster), but better than the alternatives we were exploring!! :)

Thanks again. Will run through our scenarios today on the cluster.

PS: Happy to help contribute/test the rootless task if someone can give me an overview of what is needed?

[vladaionescu]

@frafra is correct - the main issue is getting WITH DOCKER to work in rootless. I believe you can still use LOCALLY + WITH DOCKER, just not the regular FROM + WITH DOCKER, due to docker-in-docker issues with rootless. Figuring that part out is the main issue there right now, and we're open to contributions here.

[legend-zeratul]

ok, let me do some background analysis and revert with a summary - then lets see.. (tbh I might be really out of my depth here, but hey... :man_shrugging: )

[guitcastro]

Running a remote earthly-buildkitd does has this limitation as well? It´s possible to run earthly-buildkitd in rootless mode? I tried but there is a lot of configuration that must be done as root, but does could be done in Dockerfile.

[yguel]

Hi, I have the same need than in issue #1967 : Support (or document usage of) Kaniko that was closed and pointed to this issue. I cannot change the configuration of the gitlab-runner for the gitlab platform for which we need to deploy CI/CD. I decided to try earthly since we need to host the repository of our team on github and gitlab. So far, I have converted the current CI/CD and they work on github (which was already working with standard github actions). I have to decide, if I made the wrong choice to choose earthly to have a unique CI/CD for github and gitlab with unprivileged setting and switch to maintain several CI/CD.

I do not understand the workaround presented above and especially the limitations of propagation of rootless mode to all docker container used down in the CI/CD (if it is what happens). Also Kaniko seems the only other potential solution discussed so far.

Is there any progress for the support of the rootless or the kaniko solution ? Is there any pointer or shared experience that could help us running our CI on gitlab without privileged mode or to understand better how to setup a workaround or if such a workaround exists ?

[alexcb]

Note that, if you see an error such as:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: no cgroup mount found in mountinfo: unknown

it's likely you are using rootless docker, which does not currently allow the use of WITH DOCKER.