Kubernetes container

[bhack]

Support for containers running in kubernetes (pod, deployment etc..)

[chrmarti]

@markomitranic Are you connected with Remote-SSH to the SSH server or is VS Code's UI running on the SSH server?

[markomitranic]

@chrmarti Connected over Remote-SSH

[chrmarti]

Filed https://github.com/microsoft/vscode-remote-release/issues/6054 for connecting through a SSH server.

[arashnorouzi]

Hi @chrmarti, When I'm trying the "attach visual studio code" option, it opens a new vs code window and connects to the pod, but my source code is still open in the first window. Is there any way to configure local source code path for this option? Maybe I'm missing a step ? Thanks.

[chrmarti]

@arashnorouzi VS Code is not aware of any mount points inside the pod. Would you expect the existing window to be reused?

[tobru]

When using the "Attach Visual Studio Code" on a Pod it executes:

kubectl -n myns exec -it my-deployment-94bb57f74-xlkx7 -- /bin/sh -c VSCODE_REMOTE_CONTAINERS_SESSION='UUID' /bin/sh

This seems wrong as this command terminates immediately, everything after -c should be quoted. The follwowing command would be correct:

kubectl -n myns exec -it my-deployment-94bb57f74-xlkx7 -- /bin/sh -c "VSCODE_REMOTE_CONTAINERS_SESSION='UUID' /bin/sh"

Can I configure that or is this a bug?

[chrmarti]

@tobru The quoting is only needed when running the command from a shell, but we spawn the process directly, passing the parameters as an array. The part you suggest to quote is passed as a single entry in that array.

Are you seeing the command terminate immediately?

[leociociano]

I have the same problem. I followed this procedure (https://code.visualstudio.com/docs/remote/attach-container#_attach-to-a-container-in-a-kubernetes-cluster) and I get the following error. I am cluster admin. vs-error.txt

[FractalMachinist]

I'm chasing some similar issues as @tobru mentions.

Reproduction environment:

SERVER: - Ubuntu 22.04.1 LTS - Running MicroK8s v1.25.3 revision 4094 - $(microk8s kubectl version --short) => "Client Version: v1.25.3\nKustomize Version: v4.5.7\nServer Version: v1.25.3" - MicroK8s is running with an accessible, healthy container & pod & namespace - SERVER is the only MicroK8s node - SERVER is on the same network as CLIENT - MicroK8s generated kubeconfig was provided to CLIENT CLIENT: - Ubuntu 22.04.1 LTS - Kubectl is installed - $HOME/.kube/config is in place as provided by SERVER - Kubectl version is identical to SERVER's MicroK8s.kubectl version - Running kubectl in a bash terminal works correctly, including all commands which vscode-remote-release depends on (discussed below) - Specifically, kubectl interacts with SERVER correctly. - Side note - kubectl fails when CWD is within an NFS mount, which are *on* CLIENT but are not where I'm running commands. Just being thorough. - VSCODE: - Version 1.73.0 - Extensions: - Dev Containers v0.263.0 (Identical behavior tested in v0.262.3) - Docker v1.22.2 - Kubernetes v1.3.10 - `mtsmfm.vscode-k8s-quick-attach` v0.8.0 (Useful for minimum replication) - YAML, Prettier

I discovered these issues while debugging "Attach to Visual Studio".

Detailed Debugging of "Attach to Visual Studio"

# Discovering the issue Dev Containers provides a mixin to the Kubernetes extension, which is supposed to support "Activity Bar > Kubernetes > [CONTEXT] > Workloads > Pods > [POD NAME] > {CONTEXT MENU} > Attach to Visual Studio". However, this command fails in on my machine. VSCode provides a popup: > Error running command remote-containers.attachToK8sContainerFromViewlet: Command failed: kubectl get pod [POD NAME] -o json --namespace [NAMESPACE]. This is likely caused by the extension that contributes remote-containers.attachToK8sContainerFromViewlet. ...and sends a similar error to the Extension Host: >[TIMESTAMP] [error] {"message":"Command failed: kubectl get pod [POD NAME] -o json --namespace qs","stdout":{"type":"Buffer","data":[]},"stderr":{"type":"Buffer","data":[]},"code":1} remote-containers.attachToK8sContainerFromViewlet {"value":"ms-vscode-remote.remote-containers","_lower":"ms-vscode-remote.remote-containers"} # Minimizing error conditions The exit code 1 with no stderr was surprising. I pulled open `$HOME/.vscode/extensions/ms-vscode-remote.remote-containers-0.263.0/dist/extension/extension.js`, pushed it through Prettier, and started unminifying. ## Handling `remote-containers.attachToK8sContainerFromViewlet` I located the function which handles `remote-containers.attachToK8sContainerFromViewlet`. To locate this function, it is the only instance of the text "current-context", and was assigned the minified name `V9` in v0.263.2. I renamed this function `newWindowAttachedToK8sContainer`. Inside, I found a function which spawns via child_process (function originally named `ze` in v0.263.2, I renamed to `execOnArgs`), and which is used to execute `kubectl` to gather information about context and the selected pod. I added more calls to `execOnArgs` to test its behavior. My expectation for correct behavior is having similar stdout/stderr and identical exitcode vs running the equivalent command in a bash terminal, and not leading to errors while attaching to a container. ### `execOnArgs` Tests Format: \`${cmd} ${argsArray}\` All tests use the existing `env` and `output`. - Base Cases - kubectl ['config', 'current-context'] - Ends with code 0 and no stderr/stdout - bash exits code 0 and returns a string naming the current context. - This silent failure seems related to: - #7108 - possibly #5100 - possibly #4998 - kubectl ['get', 'pod', 'POD NAME', '-o', 'json', '--namespace', 'NAMESPACE'] - This is the error condition I originally discovered - Exit code 1, no stderr/stdout - bash gives a JSON description of the named pod, and exit code 0. - Test Cases - echo ['echoed_text'] - Behaves correctly - bash ['-c', 'echo echoed_text'] - Behaves correctly - bash ['-c', 'kubectl config current-context'] - Exit code 0, no stderr/stdout - Bash handles this correctly (gives stdout w/ exit 0) - Incidentally, this works in Python on the same machine - > python3 -c "import subprocess as sc;print(sc.run(['bash', '-c', 'kubectl config current-context'], capture_output=True).stdout)" - > b'microk8s\n' - bash ['-c', 'kubectl config current-context | tee /dev/null'] - WORKS! - Exit code 0, stdout indicates current context, empty stderr - Perfectly matches Bash behavior. - bash ['-c', 'kubectl get pod PODNAME -o json --namespace NAMESPACE | tee /dev/null'] - Works again - Exit code 0, stdout describes pod, empty stderr - Perfectly matches Bash behavior. ### Correcting `remote-containers.attachToK8sContainerFromViewlet` I modified `newWindowAttachedToK8sContainer`'s calls to `execOnArgs` to match the successful format (the last 2 tests). With these changes, the workflow described above ("Activity Bar > Kubernetes > ... > Attach to Visual Studio") reaches a different error related to `kubectl exec`. ## Directly testing `child_process.spawn` ### Within VSCode Digging into traces of `execOnArgs`, I found a function which calls `child_process.spawn`. This function is named `rs` in v0.262.3, and is the first instance of the text "SIGKILL". Based on this, I repeated the above tests by directly constructing calls to the `child_process` module as exposed in the extension. To summarize, the results are identical. ### VSCode in debug VS Node in gnome-terminal I built a test module to run in a pure Node environment and in VSCode. Running `node child_proc_kubectl_test.js` in a bash terminal results in the expected behavior (the current context is produced). However, if you add a breakpoint to a VSCode extension and require/run the test script, it mimics the behavior when testing in VSCode directly - `kubectl config current-context` returns code 0 and no stdout/stderr, `kubectl get pod ...` returns code 1 and no stdout/stderr. ### gnome-terminal VS VSCode Terminal Using the above test module, I ran `node child_proc_kubectl_test.js` in an instance of gnome-terminal and in the terminal within VSCode (CTRL-`). gnome-terminal produced the expected output, and VSCode's terminal did not. This is, to me, the strangest claim I make. I've embedded evidence. *Screencast of me running this test* [terminal_comparison.webm](https://user-images.githubusercontent.com/56850891/200097964-604bb3b5-6acd-494f-8c40-0cde1520260c.webm) *Test scripts* [child_proc_kubectl_test.js.txt](https://github.com/microsoft/vscode-remote-release/files/9942882/child_proc_kubectl_test.js.txt) [demo_vscode_kubectl.sh.txt](https://github.com/microsoft/vscode-remote-release/files/9942883/demo_vscode_kubectl.sh.txt) *Collected test output* [terminal_test.md](https://github.com/microsoft/vscode-remote-release/files/9942884/terminal_test.md) [vscode_test.md](https://github.com/microsoft/vscode-remote-release/files/9942886/vscode_test.md) *`diff` between test outputs* [_test.md.diff.txt](https://github.com/microsoft/vscode-remote-release/files/9942889/_test.md.diff.txt) *VSCode verbose output (elided). Produced by `$(code --verbose . |grep -vwE 'File Watcher|pointer_event_factory' > vscode.elided.log)`.* [vscode.elided.log](https://github.com/microsoft/vscode-remote-release/files/9942885/vscode.elided.log) *`diff` of environment variables between gnome_terminal and VSCode Terminal* [env.diff.txt](https://github.com/microsoft/vscode-remote-release/files/9942891/env.diff.txt) VSCode's terminals run a bash subprocess which is provided an argument `--init-file /snap/code/112/usr/share/code/resources/app/out/vs/workbench/contrib/terminal/browser/media/shellIntegration-bash.sh', which (at first glance) seems to focus on managing history and debugging tie-ins, but I didn't go into depth.

Debugging Conclusions

Some aspect(s) of VSCode cause kubectl subprocesses to give no stderr/stdout, and some subcommands/args to fail entirely, in a variety of settings:

• While attempting to "Attach to Visual Studio" re: Kubernetes pods.
• While debugging the Dev Containers extension and attempting to spawn kubectl directly.
  • eg spawn('kubectl', ['get', 'pod', '-o', 'json'])
• While debugging the Dev Containers extension and wrapping kubectl ... in a bash subprocess.
  • eg spawn('bash', ['-c', 'kubectl get pod'])
• While spawning a kubectl ... subprocess inside a Node process inside the bash subprocess backing the VSCode terminal.
  • See above screencast in 'Detailed Debugging of "Attach to Visual Studio"'

In some other settings, kubectl and VSCode get along perfectly well:

• While called directly from the VSCode bash terminal
  • eg. CTRL-` then 'kubectl get pod -o json' works fine
• While debugging the Dev Containers extension and wrapping kubectl ... | tee /dev/null in a bash subprocess.
  • See "execOnArgs Tests" in 'Detailed Debugging of "Attach to Visual Studio"'
• While spawning a kubectl ... | tee /dev/null subprocess inside a Node process inside the bash subprocess backing the VSCode terminal.
  • See above screencast in 'Detailed Debugging of "Attach to Visual Studio"'

The success and failure characteristics are identical across all these different settings. The behavior in the minimal case screencast is identical to the behavior leading to errors while running "Attach to Visual Code".

Minimal Reproducible Error

This is not the error discussed above, but is closely related. This error arises when something in VSCode calls vscode.commands.executeCommand(..., uri, ...) where uri starts with k8s-container. When that URI is interpreted by remote-containers, kubectl exec ... is launched as a subprocess to interact with the container, and fails. This same failure arises even when the errors discussed above are resolved or sidestepped.

This is, I believe, the specific error @tobru mentioned.

1. On CLIENT, follow the 'mtsmfm.vscode-k8s-quick-attach' command palette workflow
   • Select the context, namespace, pod, container, and target root to attach to.
   • This particular implementation avoids the issues in 'Attach to Visual Studio'.
2. A new VSCode window opens
   • An error modal shows "An error occured attaching to the container".
   • Behind that modal, the 'Dev Containers' terminal output shows attempts to call kubectl exec ... on the desired pod.
   • Following these attempts, an error message "Shell server terminated (code: 0, signal: null)".

shell_server_terminated.webm

Transcript of kubectl exec & errors

After verifying the error, I added some debugging to validate @chrmarti 's explanation in response - specifically, deconstructing the command into the args which get passed to `spawn`. > [60 ms] Dev Containers 0.263.0 in VS Code 1.73.0 (8fa188b2b301d36553cbc9ce1b0a146ccb93351f). > [60 ms] Start: Resolving Remote > [64 ms] Start: Run: kubectl exec -it qs-pod --context microk8s --namespace qs --container server -- /bin/sh -c VSCODE_REMOTE_CONTAINERS_SESSION='546faf54-f639-4dcc-a97d-dd5caf229e281667616251430' /bin/sh (deconstructed: 'kubectl' ["exec", "-it", "qs-pod", "--context", "microk8s", "--namespace", "qs", "--container", "server", "--", "/bin/sh", "-c", "VSCODE_REMOTE_CONTAINERS_SESSION='546faf54-f639-4dcc-a97d-dd5caf229e281667616251430' /bin/sh"]) > [67 ms] Start: Run in container: echo $PATH > [252 ms] Stop (188 ms): Run: kubectl exec -it qs-pod --context microk8s --namespace qs --container server -- /bin/sh -c VSCODE_REMOTE_CONTAINERS_SESSION='546faf54-f639-4dcc-a97d-dd5caf229e281667616251430' /bin/sh (deconstructed: 'kubectl' ["exec", "-it", "qs-pod", "--context", "microk8s", "--namespace", "qs", "--container", "server", "--", "/bin/sh", "-c", "VSCODE_REMOTE_CONTAINERS_SESSION='546faf54-f639-4dcc-a97d-dd5caf229e281667616251430' /bin/sh"]) > [259 ms] Shell server terminated (code: 0, signal: null)

Potential fixes

1. Transition to @kubernetes/client-node instead of constructing commands as strings/subprocesses
   • Pros:
     • Likely to reduce cross-platform compatibility issues (eg WSL)
     • This approach is used in 'mtsmfm.vscode-k8s-quick-attach', which is the reason that extension isn't running into errors.
     • Potentially provide marginally better tools for typechecking during development
     • Fix my issue in particular :) and several others
   • Cons:
     • Depend on @kubernetes/client-node's featureset instead of directly relating to the host kubectl's featureset
     • Invites some likelihood of version mismatches between @kubernetes/client-node and cluster API (same risk as host kubectl, but still another update to manage)
     • Potentially significant rewrite effort
2. Determine why VSCode causes wide-reaching interference with kubectl subprocesses, and fix that directly

[xgalaxy]

On Windows 10. I've been trying this attach method from the context menu but it fails to establish a connection. I get a websocket 1006 error. I'm able to pull up the remote terminal via ssh from the kubernetes plugin however. But attach doesn't work at all.

If I switch to MacOs it works perfectly.