Failed to get the container environment: error: Internal error occurred: error executing command in container: failed to exec in container

[sunshine69]

Describe the bug Bridge stop working

To Reproduce It has been working for my test application for a while. No update. However today it suddenly stop working

**Expected behavior** It works **Logs** Error: cli-client-connect-error {} Error: Failed to get the container environment: error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "0938e26c423ba7cec5bca75dc6e5d66b80886becd19bbbede62606df50ad3dda": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "env": executable file not found in $PATH: unknown\n\n at ChildProcess. (/home/stevek/.vscode/extensions/mindaro.mindaro-1.0.120210615/dist/extension.js:3:143256)\n at ChildProcess.emit (events.js:315:20)\n at ChildProcess.EventEmitter.emit (domain.js:467:12)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12) 2021-07-02T02:35:50.247Z | Connect (bridge-k8s-test | ERROR | Error: connect-error {} Error: Failed to get the container environment: error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "0938e26c423ba7cec5bca75dc6e5d66b80886becd19bbbede62606df50ad3dda": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "env": executable file not found in $PATH: unknown\n\n at ChildProcess. (/home/stevek/.vscode/extensions/mindaro.mindaro-1.0.120210615/dist/extension.js:3:143256)\n at ChildProcess.emit (events.js:315:20)\n at ChildProcess.EventEmitter.emit (domain.js:467:12)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12) 2021-07-02T02:35:50.255Z | Common Extension Root | ERROR | Error: connect-service-task-terminal-error {} Error: Failed to establish a connection. Error: Failed to get the container environment: error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "0938e26c423ba7cec5bca75dc6e5d66b80886becd19bbbede62606df50ad3dda": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "env": executable file not found in $PATH: unknown\n\n at /home/stevek/.vscode/extensions/mindaro.mindaro-1.0.120210615/dist/extension.js:3:77473\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:93:5) **Environment Details** Client used (VS Code/Visual Studio): Client's version: Version: 1.57.1 Commit: 507ce72a4466fbb27b715c3722558bb15afa9f48 Date: 2021-06-17T13:26:56.255Z Electron: 12.0.7 Chrome: 89.0.4389.128 Node.js: 14.16.0 V8: 8.9.255.25-electron.0 OS: Linux x64 5.12.13 **Additional context** Add any other outputs from the clients or context you would like to share.

[sunshine69]

This error is an example of bad error reporting.The error points nothing for the user to debug.

However if setup in a new virtual machine all from scratch, it did not happen.

IN the machine gets the issues I tried to move away the whole ~/.vscode and ~/.config/Code so it would be the same as from scratch. Not fixed!.

Same machine, switch user to another user and run, it works.

The output of the env command in a both working and not working case is identical. And the machine which work is exactly the same ( running using the same system image and software)

I am sort of stumped.

[sunshine69]

All of that points to a question, Which system components that affect the plugin to works ? Can you guys please help so we can know a bit more about its behavior.

[daniv-msft]

Thanks @sunshine69 for logging this issue, and sorry you encounter it. We're looking into it, and still need to look further to be certain of the root cause. Our hypothesis is that the pod you try to debug contain multiple containers, and that for some reasons we pick the wrong one. This wrong one likely is based on a minimal Docker image that doesn't contain the "env" utility we rely on.

If you look at the pod targeted by your service, does it contain multiple containers? Could you please try to run kubectl exec {podName} -c {containerName} -- env for each of the containers in the pod, to see if you can reproduce the error and validate our hypothesis?

To answer your question, apart from the VS Code extension, we also download binaries we require. You can find them here: %AppData%\Code\User\globalStorage\mindaro.mindaro\file-downloader-downloads Finally, we are dependent on your current kubeconfig's context, as this is what we rely on to connect to the cluster.

[sunshine69]

Ah yeah my pod is minimum so no env command. However the things is why it happened only on my current linux user account but not other (created from scratch in the same machine). There must be something in the environment caused its behaviour but compare env output is identical.

If it is because of the minimum docker image lacking cmd env it should happen all the time.

I know the file-downloader-downloads and I already tried to remove the whole config and cache dir ~/.vscode and ~/cache/Code, etc but still exists.

The kube config dir is identical for both case, work and not work.

Or there is something installed locally in my account that other not have, such as the azcli and some python module but I don't see why the bridge uses it all all.

I will try your tips soon, but you can see in which case the bridge needs to exec inside container ? and what not ..

Ta

[sunshine69]

Here is the outcome, as expected my CT does not have env. It has only one container

kubectl exec test-bridge-k8s-webservice-0  -- env
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "a0285bc8557dfee24ac1a2bb96ae2b444447a953c503d8e03cbe9f1ca7da5a47": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "env": executable file not found in $PATH: unknown
stevek@macbook-work 11:20 ~> 

But the container hash is different form the vscode error output below

EndpointManager came up successfully.
Failed to get the container environment: error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "ff77077fdaa4bd26c181ed6a87f4d9cb59e9aabbaf5c62387af01aadb99913b1": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "env": executable file not found in $PATH: unknown
Stopping workload and cleaning up...
Failed to establish a connection. Error: Failed to get the container environment: error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "ff77077fdaa4bd26c181ed6a87f4d9cb59e9aabbaf5c62387af01aadb99913b1": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "env": executable file not found in $PATH: unknown

Looks at the message format similar obviously the bridge try to exec into a container (some not sure what) and got error.

Question is:

• why it works for other case
• why it needs to exec at all. It should get the k8s manifest which already hold all information in order to replicate a proxy pod.
• what it wants to extract from by execting ...

[daniv-msft]

Thanks @sunshine69 for running the command. We try to exec into the container in order to get the environment variables that are available in this container. We do this because, when we run your microservice locally, we try to replicate conditions similar to the cluster, and your microservice might require some of the environment variables only available from inside the cluster.

We're now working on a fix (adding @amsoedal who works on it) that would not inject any environment variables if we can't technically retrieve them from the container. We hope to provide this bug fix soon.

Regarding your question about why it works in the other case, the only explanation I have is that it might be using an older version of our code. Previously, we were only retrieving environment variables from the container after we injected our own Docker image in it. This gave us the guarantee that we would always be able to get the environment variables, but came with other issues.

[UnwashedMeme]

Regarding your question about why it works in the other case, the only explanation I have is that it might be using an older version of our code. Previously, we were only retrieving environment variables from the container after we injected our own Docker image in it. This gave us the guarantee that we would always be able to get the environment variables, but came with other issues.

That's a big, notable, change that just broke our code too. This would be good to highlight as a requirement of this extension.

[daniv-msft]

@UnwashedMeme Sorry for the disruption, we did underestimate the impact this change would have. We didn't expect users to have minimal Docker images that wouldn't allow us to retrieve the environment variables.

Our plan is to fix this bug by ignoring the environment variables injected in the container if we cannot retrieve them. We expect that this should fix the issue for most users impacted, while not adding a new requirement to use Bridge.

[UnwashedMeme]

@daniv-msft By "ignoring the environment variables injected in the container" we're going to lose some of the utility of this.

Suggestions:

• do the injection first (I don't know what the issues you faced with this were)
• Catch & Clarify this error message so users have a better idea what's up.

I think a common pattern, at least in the go community is to build the binary and then add that to a FROM scratch container. I'm looking at building extra images we can switch to prior to injection.

[daniv-msft]

@UnwashedMeme Thank you for sharing your experience on how Go developers usually create images. Yes, we will make sure that, when this case happens, we give a message so that the user knows we couldn't retrieve the environment variables (+ @amsoedal who is the one implementing this on our side) and suggest what to do next.