Kubernetes container

[bhack]

Support for containers running in kubernetes (pod, deployment etc..)

[bhack]

November plan was finalized. No Kubernetes initial support this month.

[bhack]

Any Kubernetes support again in the Jan 2020 dev iteration also if this ticket is in the top 2 community priorities for containers section: https://github.com/microsoft/vscode-remote-release/issues/2075

[bhack]

Explore support to attach to Kubernetes

is in the Feb plan https://github.com/microsoft/vscode-remote-release/issues/2329

[chrmarti]

The latest Remote-Containers (0.107.0) for VS Code Insiders now supports attaching to a container in Kubernetes. There is a context menu action Attach Visual Studio Code on pods in the Kubernetes Explorer:

Gif is too large to post here: https://twitter.com/christof_marti/status/1237395371241791489?s=20

[lumaxis]

I had pinged @chrmarti on Twitter about this already bug adding a note here too: Currently, the extension requires the Kubernetes container to be running as root which many services will likely not, due to the security concerns that come with that. Would be great if it also worked with a non-root user.

[chrmarti]

@lumaxis I have a fix for running as non-root queued for the next release. Should be out for VS Code Insiders sometime next week.

[lumaxis]

Great! I'll keep an eye out for the change and give it another spin then 👌🏼

[bhack]

@chrmarti What is the next step? Do you plan to support run?

[chrmarti]

@bhack That could be a next step. I imagine that you would usually have several containers deployed and the DevContainer would be one of them. In that case run might only work well when the DevContainer only has dependencies on the other containers and not the other way around.

[bhack]

@chrmarti Probably Yes. For more complex scenario we could support apply. I don't know if we could control kubectl apply -k so that Kustomize could handle more complex configurations.

[TripleEmcoder]

Judging from the error message I get when trying to attach to a Windows container, this isn't supported yet, is it?

[chrmarti]

@TripleEmcoder Correct, Windows containers are not supported currently.

[erihanse]

Nice job! Can we also support the scenario where read-only root filesystems are enabled?

[itssimon]

I can't get remote workspaces to work. I can open a single remote folder, but adding more folders to the workspace breaks it. Folders are showing in yellow with an exclamation mark next to it and no content.

[chrmarti]

Created new issues for the above mentioned request (#2755) and bug (#2756). Thanks.

[kwlzn]

does Remote-Containers support attaching to remote kubernetes pods via kubectl? when I attempted to use it, it seemed to require a local docker daemon.

we'd like to connect to pods in large compute clusters like e.g. Google's GKE - wasn't able to figure out how to get that working. any pointers?

[pchico83]

@kwlzn do you know. this extension? https://marketplace.visualstudio.com/items?itemName=okteto.remote-kubernetes

[chrmarti]

@kwlzn You need Microsoft's Kubernetes extension and the Remote-Containers extension installed to attach to a container in Kubernetes with the work being done here.

See the VS Code 1.44 Remote Release Notes for a screenshot: https://github.com/microsoft/vscode-docs/blob/master/remote-release-notes/v1_44.md#kubernetes-support

[bhack]

@chrmarti Any idea for April? Is there any proposal for attach/run? Cause run still doesn't support Manifest https://github.com/kubernetes/kubernetes/issues/63214. Instead apply It work also on complex config cases cause it sopport also kustomize parameter.

[bhack]

Sorry I meant apply. It was a typo.

[chrmarti]

@bhack Nothing is planned for April.

I wonder if it makes sense for us to favor one way of creating a container over another or if we should try to find a way to generically support user-defined commands to create containers.

Also: Where does that information on how to create a container come from? Would that be in user settings or would there be a locally checked out repository with the information on how to get to a container and that container could then contain (or have mounted) the source repository (which can be different)?

With Docker we set things up in a way that works well locally where the same devcontainer.json contains information on how to create a container and how to configure it once connected.

[bhack]

I think that we need to maintain all the Docker features related to devcontainer.json but we could add a push step cause often the remote container need to (optionally) pull the image from a remote registry in Kubernetes in the case we manipulate the Dockerfile (currently we monitor Dockerfile but we propose only a build step).

Then I think that we could expand devcontainer.json with a the deployment command in as an initial feature. So that user can users could choice if the want to deploy with helm, kubectl, etc.. command.

Then later we could support of kind of image builds step that are not strictly related to docker build (i.e. remote kokoro etc...)

[PavelSosin-320]

@bhack There is no "remote" registry. I deployed the registry image to my Docker engine in 2 minutes and use it for docker service deployment. All used images can be replicated from the central repository to local and back manually at the cost of 5 minutes per image. docker-compose push and docker stack deploy are unaware which repository is used - it is simple ARG. It makes the Development environment fast - no need to publish tag for every build and testing & production environment stable - local registry tags can reference vscode version and back.

[bhack]

E.g. But how you handle that case when your Docker engine infrastrcture It Is running under a remote kubernetes cluster where there is the hardware that you want to develop on like ML/Deep learning development where you want to develop GPU code on a remote GPU container/node provisioned by Kubernetes.

[wstrange]

Not sure if we are talking about the same thing, but we are experimenting with using kaniko in conjunction with vscode remote - so the pod does not need access to the docker socket.

[bhack]

@wstrange We have a Dockerfile monitoring in this extension that propose you to rebuild the image and restart the container if needed. I cannot see the internals cause the extension is closed source but how do you propose to handle the interaction of this machinery with Kaniko?

[wstrange]

I think you are talking about the Dockerfile that defines the pod where vscode is running itself? I dont think Kaniko will solve that problem.

We bundle up all the tools we need for development (skaffold, kustomize, etc.) into the pod, and run skaffold inside the pod (we install the google cloud code plugin in the remote side). Skaffold uses the kaniko builder - so we do not need access to the docker socket.

[bhack]

Yes I am talking about the Dockerfile that is referenced by the .devcontainer.json on which this extension is interacting on.

[PavelSosin-320]

I don't understand something; dickerfile which defines a pod? Dockerfile dependent on hardware? In Kubernetes pod which contains hardware-dependent images can be scheduled to the nodes with such hardware by the label. Hardware drivers can be installed on the node because they are hardware-dependent. How Docker container, i.e. subprocess can communicate with the hardware directly bypassing pod and node isolation? Only Windows containers on the Windows machine can ignore isolation. Do you mean something like NVIDIA container which doesn't use any Kaniko?

[bhack]

Currently the extension is supporting only attach so it doesn't care how you have deployed the pod. Here we are discussing the next iteration steps.

[bhack]

@chrmarti Do you think that we could start to control the deployment in this month iteration?

[bhack]

I think that what we mentioned at https://github.com/microsoft/vscode-remote-release/issues/12#issuecomment-613359085 could be the next step.

[chrmarti]

@bhack We don't have this planed for the June milestone. The main challenge from my point of view is that there are several commands for deployment, each with its own options to affect the outcome. It is not clear to me that picking one command and a subset of options will get this to work for most users. Offering all deployment commands and all their options on the other hand seems overly complex to implement and use.

What if you could configure the commands used to set up and connect to a DevContainer? The commands can use kubectl or any other locally available CLI.

A simple way of doing this would be to have a single command that sets up the container (if not set up already) and ends with a /bin/sh session that Remote-Containers can use to install the VS Code Server.

[bhack]

@chrmarti So what do you think we will have? As we are monitoring the Dockerfile do you think that we need an user pre-deployment command (i.e. for docker build and push, remote build and push, etc..) and an user deployment command? Do you want to pilot generic executables or do you want to interact with the cluster using the JavaScript client library or a mixed approach?

[chrmarti]

Trying to come up with a simple example. You could also run separate script files for each of these:

{
    "start": "kubectl create deployment --image=nginx nginx-dev ; kubectl wait --for=condition=Available deployment.apps/nginx-dev",
    "attach": "kubectl exec -i deployment.apps/nginx-dev -- /bin/sh",
    "rebuild": "kubectl delete deployment nginx-dev",
    "stop": "kubectl delete deployment nginx-dev"
}

Only "attach" would be mandatory, the other commands would be optional. This configuration would have to be available locally either as user data or checked out from SCM.

Detecting changes in a Dockerfile or other files would require additional configuration.

[bhack]

I think this command could be ok but are you sure that for the kubectl case you don't want to use the client library instead of interacting with the exec?

[chrmarti]

Using the CLI has the advantage that we can leave the choice of deployment and other details to the user. We might provide examples of how you can set things up, but you could then still tune them to your specific needs. I haven't explored this in much depth yet though, does that sound like it might work for what you plan on doing?

[bhack]

This really depend on our use case scenario. I meant if we think that 80% of the targets commands Is kubectl we could integrate it with the library for a small subset of commands. Then we could let the user to override it like in your example. For that case we will not have so much control on command outputs/returns but It could be ok for custom cases to let the user to just check the log if something goes wrong.

[zywillc]

any plan or progress on this request?

[Phantomape]

Attach Visu

Hi, I was using Attach Visual Studio Code in my daily development, it worked perfectly. However, one day I decided to use Intellij IDEA in the pod terminal(more comfortable using IDEA to develop Java program :) ), and I found that X forward is not enabled by default. I manually did a ssh -Y <user>@<host> login to the remote host and set the DISPLAY env var in the pod, and then I was able to launch Intellij. Is there a way to enable X forward and use GUI programs in the pod after I attached to the remote container? Post the same question in https://github.com/Azure/vscode-kubernetes-tools/issues/849 but it seems that I dug into the wrong place to ask.

[gaocegege]

Hi, is there any progress?

There are lots of users (internally) who want to develop deep learning models on Kubernetes with the help of GPUs. I have to help them install sshd in the docker container then run it as @bhack said. I am wondering if you have a plan to support Kubernetes in the near future, or I need to consider switching to other possible solutions like okteto.

I'd appreciate it if anyone could help me.

Thanks :clinking_glasses: :beers:

[chrmarti]

@gaocegege There is an Attach Visual Studio Code action on pods in the Kubernetes Explorer, so there is no need to set up sshd. Can you give that a try?

[gaocegege]

Thanks for your reply!

It works. But I am wondering if we have the plan to support other commands like launch and so on, just as remote-container docker does.

BTW, is there any plan to open-source remote container extension?

[chrmarti]

We are tracking supporting additional commands in this issue, but there is currently no timeline.

There are no plans to change the license. (FAQ on licensing: https://code.visualstudio.com/docs/remote/faq#_license-and-privacy)

[gaocegege]

OK gotcha, thanks for your reply!

[markomitranic]

@chrmarti Thanks for the explanation, I've seen the same example somewhere in the documentation. However my context menu does not offer the option you refer to. Why would that be?

[chrmarti]

The docs show it under Pods: https://code.visualstudio.com/docs/remote/attach-container#_attach-to-a-container-in-a-kubernetes-cluster

Is the tree node you are opening the context menu on a pod?

[markomitranic]

Hey @chrmarti thanks for your swift response. The Pods subtree behaves the same as if targeting the pod from the deployments section. I get the same context menu as in the screenshot.

[chrmarti]

@markomitranic Not sure this is obvious: You need the Remote-Containers extension in addition to the Kubernetes extension. Make sure you have that installed too.

[markomitranic]

@chrmarti Haha good point, of course, it is installed. In fact, what you are looking at is not my local vscode at all, it is a remote instance SSH running on a VPS. My local computer is not used for running code or connecting to the cluster at any point.