Docker support

[configurator]

The default image(s) should include some sort of docker support, so I can run docker build, as well as docker start to run either my own service (and expose ports as usual) or its requirements.

[svenefftinge]

That's unfortunately not possible to do as it would require running the workspace containers in privileged mode. But we should have a command line util that allows to build and run docker images externally as a service.

[dimensi0n]

Any news about this command line util ?

[jankeromnes]

@dimensi0n we haven't looked at implementing this yet, but now I'm thinking that maybe we could somehow interface a local docker CLI with our image-builder (since I can already make it build stuff by adding a Dockerfile to my .gitpod.yml, I don't see why I couldn't also do the same via an in-workspace docker CLI).

Maybe this could also somehow be a way to improve Gitpod's repository setup automation workflow (where currently I need to commit a Dockerfile, push it to a branch, manually create a new workspace for that branch, test it, and return to my first workspace to iterate in a relatively tedious way).

[JesterOrNot]

Just spitballing but could you install docker in the default Gitpod image and then add the gitpod user to the docker group? I think that it would allow users to test Dockerfiles from the command line.

[jankeromnes]

Thanks for the suggestion @JesterOrNot. Unfortunately, I believe that from a security standpoint, adding user gitpod to the docker group is akin to giving users sudo rights. See also https://github.com/gitpod-io/gitpod/issues/755.

However, maybe it's possible to install the docker CLI in the default Gitpod image, and configure it to use Gitpod's image-builder (you can already make it build anything by adding a Dockerfile to your project, so CLI would just be a more convenient access) instead of building containers locally (which would be docker-in-docker, and require some sort of risky security trade-offs).

[JesterOrNot]

Heres another idea you know google cloud shell? well there is a vscode extension that allows you to access it remotely it's free and has docker installed in it could we either set it up as a remote client or have the extension pre-installed?

[JesterOrNot]

Here is a repo with the Docker CLI installed in it can someone help me get it hooked up to the gitpod image builder? or maybe just send a PR https://github.com/JesterOrNot/Gitpod-Docker

[jankeromnes]

Thanks a lot @JesterOrNot!

I'm not sure how to use Gitpod's image-builder API. @32leaves or @geropl do you have any insights here? (We're trying to make image-builder build a Dockerfile, not by committing to a repo and opening it in Gitpod, but by directly sending the Dockerfile to image-builder.)

[JesterOrNot]

Any updates per the status of this issue?

[meysholdt]

Hi @JesterOrNot ,

not much of an update, unfortunately, but more details to share.

The challenge here is that the standard docker daemon requires root privileges and in the Gitpod workspaces we don't have root privileges.

Things you could do:

1. bring-your-own-docker-engine. See here for an example. This means you run a docker engine on your own infrastructure and connect to it from a docker workspace.
2. Get Rootless Docker working. In theory this should work in GItpod, but in practice something doesn't work yet. The following Dockerfile from @geropl may by a good stating point:

   
   USER gitpod

Rootless Docker

gets installed to /home/gitpod/bin

RUN curl -sSL https://get.docker.com/rootless | sh

It requires the following env vars:

ENV XDG_RUNTIME_DIR=/tmp/docker-33333 ENV PATH=/home/gitpod/bin:$PATH ENV DOCKER_HOST=unix:///tmp/docker-33333/docker.sock

Things the Gitpod team can be doing:
3. Allow triggering Gitpod's image builder form a workspace. However, even if we do this, this will never be a replacement for full docker support. It will only be to test if the dockerfile configured in .gitpod.yml works. And for security reasons, we can't allow `docker run` in the image builder. 
4. Improve sandboxing of workspaces and allow root in workspaces. 

But we don't have a time plan for this yet, so playing with option (1) and (2) will be well worth it.

[JesterOrNot]

I'll get started with option 2!

[JesterOrNot]

I think the best route to official docker support is likely for us to support other base images because a lot of the "rootless" docker images are not Debian/Ubuntu based.

[brandonkal]

Have you considered podman rootless and setting alias docker=podman?

[JesterOrNot]

@brandonkal I'll take a look. progress can be tracked at https://github.com/JesterOrNot/gitpod-podman

[brandonkal]

@JesterOrNot I just tried using podman and it is broken Dockerfile

FROM nginx:mainline-alpine
RUN rm /etc/nginx/conf.d/*
ADD hello.conf /etc/nginx/conf.d/
ADD index.html /usr/share/nginx/html/

gitpod /workspace/pulumi-gitpod/docker $ podman build .
cannot clone: Operation not permitted
Error: could not get runtime: cannot re-exec process

I've also tried k3s in rootless mode, which also fails.

curl -sfL https://github.com/rancher/k3s/releases/download/v1.0.0/k3s -o k3s
gitpod /workspace/pulumi-gitpod $ chmod +x k3s
gitpod /workspace/pulumi-gitpod $ ./k3s server --rootless
INFO[0000] Preparing data dir /home/gitpod/.rancher/k3s/data/2492a28b74abfbdbd3f943c8109a93e979669647bbbe95eadae43b60c667df69 
FATA[2019-12-09T19:56:12.743952707Z] failed to start the child: fork/exec /proc/self/exe: operation not permitted

Running k3s inside gitpod would be very cool as it would mean I could easily share a kubernetes reproduction.

Some info that may help: https://rancher.com/docs/k3s/latest/en/configuration/#rootless-experimental https://github.com/rootless-containers/rootlesskit

And the workspace I have been experimenting in https://github.com/brandonkal/pulumi-gitpod

[JesterOrNot]

@brandonkal AFAIK we need to extend our images from gipod/workspace-full or gipod/workspace-full-vnc EDIT Nvm

[JesterOrNot]

See https://github.com/docker/for-linux/issues/885 this is probably why rootless docker is not working

[Kreyren]

rootless docker is not working

Is gitpod's docker running as root?

[jankeromnes]

Is gitpod's docker running as root?

Yes, I believe that's the case in Gitpod's Kubernetes infra.

[geropl]

@JesterOrNot We tried different tools:

• podman
• buildah
• rootless docker They all require privileged (or at least unshare), which we do not allow at the moment.

[felipecrs]

To address this issue, without having to do anything hacky, the perfect solution would be the sysbox.

If we had this kind of setup in the Kubernetes cluster, we could not only run Docker in Docker without --privileged but also do some other amazing things that seems to fit exactly in Gitpod's case.

Nestybox enables you to run system level workloads in Docker containers without using: Unsecure privileged containers Complex Docker run configurations Container image restrictions

This means the ability to run Docker in Gitpod, plus also other services with systemd (no more hacks in entrypoint or the Dockerfile).

[brandonkal]

So a different container runtime. Looks interesting @felipecassiors but there is no license that I can see.

[configurator]

It also seems to be brand new, not yet v1, and gives some seccomp permissions to containers that I personally wouldn't feel comfortable with without a thorough security review.

[JesterOrNot]

@csweichel is the one to render judgment I think, we are already looking into micro vms that could solve this issue either way

[felipecrs]

So a different container runtime. Looks interesting @felipecassiors but there is no license that I can see.

They're not charging, but this doesn't mean they can't charge in the future. However, nothing that an agreement can't solve, don't you think? Maybe the people behind Nestybox praise open source as much as Gitpod, and would like to help with.

[Kreyren]

Any update on this?

[Kreyren]

Relevant: https://github.com/docker/compose/issues/7244#issuecomment-625834398

Development of rootless docker

[ceefour]

Need this too!

[felipecrs]

Sysbox is now open-source. I believe you should revisit this alternative, as long as your VMs runs an Ubuntu OS newer than 18.04.4 it's able to run the containers with sysbox.

[gruberdev]

Sysbox is now open-source. I believe you should revisit this alternative, as long as your VMs runs an Ubuntu OS newer than 18.04.4 it's able to run the containers with sysbox.

It's important to note some cloud-oriented build images of Ubuntu 19.04 with older kernels don't have shiftfs out of the box and could be a potential source of problems even if they meet the cited requirements.

[ceefour]

I really hope this will be implemented soon.

This is a major required feature!

It's quite painful currently because when I want to do a build "locally" or experiment with Dockerfiles, I have to checkout the project "really locally", setup stuff etc. and wait a lot (due to slower internet compared to Gitpod's connection). This significantly slows down Docker-built projects.

[ceefour]

When Gitpod supports docker and hopefully docker-compose too, that'd be awesome!

[brandonkal]

Unsubscribed due to +1 noise. Please mention if you need to reach me.

[kholisrag]

any ETA about this one?

[corneliusludmann]

No ETA yet but I can give you hope that in the not too distant future there will be some news on that topic.

[jankeromnes]

FYI we're doing a YouTube livestream about this feature in about ~2.5 hours: https://www.youtube.com/watch?v=l4I2TVAnBuw

[jankeromnes]

This now works in Gitpod's (internal) staging environment, and should be included in Gitpod's next production deployment (as a Beta feature that can be enabled in Settings).

We will close this issue when docker was confirmed as working in a production Gitpod workspace. 🙂

[vfarcic]

+1

[jankeromnes]

Successfully deployed and confirmed working in production (as a Beta feature)! 🎉

Here is how to get docker / docker-compose in your Gitpod workspace:

1. In https://gitpod.io/settings/ enable "Feature Preview" (but first read the description carefully)

2. Then create a new Gitpod workspace for any repository or branch

3. In that workspace, start the Docker daemon with sudo docker-up

4. In a new Terminal, you should now be able to run docker images, docker build ., docker-compose up, etc.

Note that this is currently a Beta feature, and it may have a few rough edges (e.g. it may impact workspace start-up times -- if in doubt, you can always disable "Feature Preview" again).

If you notice anything strange, please report it: You can add a comment here, or file a new issue, or post feedback in https://community.gitpod.io

Enjoy! We're super proud to be able to bring you this highly requested feature, and we hope it will bring you a lot of value and joy. (We personally can't stop using it & finding new use cases for it. 😄)

[gengjiawen]

Thanks for the hard work. I just tried, it all works :) 👍

[davemecha]

This is so awesome to have Docker support. Thanks. 👍

Unfortunately I have a problem using Docker with setting up a development Kubernetes cluster for a out of the box Knative development setup. I still run into privilege errors. These are different issues, depending what kind of solution I try.

I documented my issues in the community https://community.gitpod.io/t/knative-development-on-gitpod/2814

I'm not sure if this is a topic for here, there or if I should file new issues.