search

loft-sh / devpod

Codespaces but open-source, client-only and unopinionated: Works with any IDE and lets you use any cloud, kubernetes or just localhost docker.
https://devpod.sh
Mozilla Public License 2.0
8.78k stars 331 forks source link

Kubernetes provider broken in 0.3.5 #646

Closed alexef closed 1 year ago

alexef commented 1 year ago

What happened?
same provider used to work in 0.3.4, after upgrade it fails with:

Run init provider command: # Check if kubernetes is reachable
${KUBECTL_PATH} get pods --kubeconfig=${KUBERNETES_CONFIG} --namespace=${KUBERNETES_NAMESPACE} --context=${KUBERNETES_CONTEXT} >/dev/null 2>&1
if [ $? -ne 0 ]; then
>&2 echo "Seems like kubernetes is not reachable on your system."
>&2 echo "Please make sure kubectl is installed and working."
>&2 echo "You can verify if kubectl is working correctly via: ${KUBECTL_PATH} get pods --kubeconfig=${KUBERNETES_CONFIG} --namespace=${KUBERNETES_NAMESPACE} --context=${KUBERNETES_CONTEXT}"
exit 1
fi
Seems like kubernetes is not reachable on your system.
Please make sure kubectl is installed and working.
You can verify if kubectl is working correctly via: /usr/local/bin/kubectl get pods --kubeconfig=/Users/alex/.kube/config --namespace=devpod --context=testing20
exit status 1
init
github.com/loft-sh/devpod/cmd/provider.initProvider
/Users/runner/work/devpod/devpod/cmd/provider/use.go:191
github.com/loft-sh/devpod/cmd/provider.setOptions
/Users/runner/work/devpod/devpod/cmd/provider/use.go:164
github.com/loft-sh/devpod/cmd/provider.(*SetOptionsCmd).Run
/Users/runner/work/devpod/devpod/cmd/provider/set_options.go:70
github.com/loft-sh/devpod/cmd/provider.NewSetOptionsCmd.func1
/Users/runner/work/devpod/devpod/cmd/provider/set_options.go:40
github.com/spf13/cobra.(*Command).execute
/Users/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
/Users/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1044
github.com/spf13/cobra.(*Command).Execute
/Users/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:968
github.com/loft-sh/devpod/cmd.Execute
/Users/runner/work/devpod/devpod/cmd/root.go:90
main.main
/Users/runner/work/devpod/devpod/main.go:8
runtime.main
/Users/runner/hostedtoolcache/go/1.20.5/x64/src/runtime/proc.go:250
runtime.goexit
/Users/runner/hostedtoolcache/go/1.20.5/x64/src/runtime/asm_arm64.s:1172

What did you expect to happen instead?

kubernetes provider to work

How can we reproduce the bug? (as minimally and precisely as possible)

My devcontainer.json:

{
    "name": "...",
    ...
}

Local Environment:

DevPod Provider:

Anything else we need to know?

alexef commented 1 year ago

tried with the 0.3.6 alpha release, and outcome is the same: 

Run init provider command: # Check if kubernetes is reachable
${KUBECTL_PATH} get pods --kubeconfig=${KUBERNETES_CONFIG} --namespace=${KUBERNETES_NAMESPACE} --context=${KUBERNETES_CONTEXT} >/dev/null 2>&1
if [ $? -ne 0 ]; then
>&2 echo "Seems like kubernetes is not reachable on your system."
>&2 echo "Please make sure kubectl is installed and working."
>&2 echo "You can verify if kubectl is working correctly via: ${KUBECTL_PATH} get pods --kubeconfig=${KUBERNETES_CONFIG} --namespace=${KUBERNETES_NAMESPACE} --context=${KUBERNETES_CONTEXT}"
exit 1
fi
Seems like kubernetes is not reachable on your system.
Please make sure kubectl is installed and working.
You can verify if kubectl is working correctly via: /usr/local/bin/kubectl get pods --kubeconfig= --namespace=devpod --context=testing20
exit status 1
init
github.com/loft-sh/devpod/cmd/provider.initProvider
/Users/runner/work/devpod/devpod/cmd/provider/use.go:191
github.com/loft-sh/devpod/cmd/provider.setOptions
/Users/runner/work/devpod/devpod/cmd/provider/use.go:164
github.com/loft-sh/devpod/cmd/provider.(*SetOptionsCmd).Run
/Users/runner/work/devpod/devpod/cmd/provider/set_options.go:70
github.com/loft-sh/devpod/cmd/provider.NewSetOptionsCmd.func1
/Users/runner/work/devpod/devpod/cmd/provider/set_options.go:40
github.com/spf13/cobra.(*Command).execute
/Users/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
/Users/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1044
github.com/spf13/cobra.(*Command).Execute
/Users/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:968
github.com/loft-sh/devpod/cmd.Execute
/Users/runner/work/devpod/devpod/cmd/root.go:90
main.main
/Users/runner/work/devpod/devpod/main.go:8
runtime.main
/Users/runner/hostedtoolcache/go/1.20.5/x64/src/runtime/proc.go:250
runtime.goexit
/Users/runner/hostedtoolcache/go/1.20.5/x64/src/runtime/asm_arm64.s:1172

(note that if I run the command by hand, it shows no pods but it works, $? is 0)

FabianKramm commented 1 year ago

@alexef thanks for creating this issue! With v0.3.6, can you do a devpod provider update kubernetes and then retry?

alexef commented 1 year ago

it still doesn't work. if I create the provider on 0.3.4 then any later version will complain about "kubectl not found" - even if I pass the full path, /usr/local/bin/kubectl, it still doesn't work. with 0.3.5 and 0.3.5 I cannot create the kubernetes provider as it fails to initialize it (with the error in this issue).

alexef commented 1 year ago

Later Edit: tried with latest release (20h ago), and I am able to create the provider.

However, when I try to create the workspace, it fails once again with:

[09:59:12] debug Acquire workspace lock...
[09:59:12] debug Acquired workspace lock...
[09:59:12] info Creating devcontainer...
[09:59:12] debug Inject and run command: '/Applications/DevPod.app/Contents/MacOS/devpod-cli' agent workspace up --workspace-info 'H4sIAAAAAAAA/9RU0XKbOBT9Fz2D5di7TcsbwYrLBAODhGfbF0ZBcqyJjBgkaGYz/vcdYRqTWVM3j30EnXt17jnn6hX8UM2zrmnJgfcKBAMeYFq7z+0jbypuuAYOaC/8dhnf0VYaly13y8Xn5V+f6I3Lbm8fb4EDdkoy3gAPwFzzRkMq+QucMd7VisFSVYa/GA2HDvCNgYb/u7puVCf6Xq+gogcOPPDuXNVGqErb41WIHwocfkf2o6OyteCb+VrYCTRv0lMrBjzTtPzogDD2AxJuQ/KtIOEGJTmxlRehD/kdymJEEC6CJL4P17+JJOgfMqZjuDaielrMpziNymN/g3DqB+/mOWl4ufp4dMCBlntRWS+PDhCMj3TrdKkYB0cHaNU2J7+lKqm8v+SWNcXaYfFlw6mVmYgD14YeauCBxXyxdOef3cUXMr/1/v7i3Sy+AwdIqk2uLaUpxGB/P0vvv72BPvHKvBGyQjUtt/ZTs7fE/LqWouxJaLjiXarYjNY1DGyzymi4oWWC4Ukdt5TWc6Z+VFJRlmcR8MDemFp7ED4Js28fZ6U6QKl2xtX7oQo2XHKqbQiHQtjNZ8vZJ+AA/sLLk6SsEV2v1bsUjj6815+k7c/SSOD0BgwrdjawF5XHo6Nh5rNA47Q8tkKyZ2HSRnRC8qde4h2VupeJN1poq8RWyfbAsfj3LfyWtCqf+xWyCSmlSM5Lc3nh++CMAjPasrs8jFYPISnCjb9Gk2vwBouTFSowilBAkuw6PM3CbRihNVqNU/9zzl/XZggneRYg/OtrigylCQ5Jkn2bBAZRjgnKiiyJpmcMMuQTdHlPBy8v1n34ofqKohRlVxQfQNdF+OCzF5CoSH3ydUz2HO0/97V0QOTfoWhap99Lbpqsio0fh/cIk4KgTRr5ZNqkdBsUfhAgjItNsprGXXcRo2wbBsh2S/J42kJMksxfoyKIfDzV7Xj8DwAA//8BAAD//2uP9UwHCAAA' --debug
[09:59:12] debug Execute command locally
[09:59:12] info Use /Users/alex/.devpod/agent/contexts/default/workspaces/dss-kubernetes as workspace dir
[09:59:12] info Delete old workspace 'dss-kubernetes'
[09:59:12] info Removing DevPod container from server...
[09:59:12] info Use Kubernetes Namespace 'devpod'
[09:59:12] info Use Kubernetes Context 'testing20'
[09:59:12] info Run command: kubectl --namespace devpod --context testing20 get pvc devpod-dss-kubernetes-default-a346ba601a0-3aa81 --ignore-not-found -o json
[09:59:12] info Removing container: find dev container: exec: "kubectl": executable file not found in $PATH
[09:59:12] debug Created logger
[09:59:12] debug Received ping from agent
[09:59:12] debug Workspace Folder already exists
[09:59:12] debug Use Kubernetes Namespace 'devpod'
[09:59:12] debug Use Kubernetes Context 'testing20'
[09:59:12] debug Run command: kubectl --namespace devpod --context testing20 get pvc devpod-dss-kubernetes-default-d3f328346a1-d77b7 --ignore-not-found -o json
[09:59:12] info find dev container: exec: "kubectl": executable file not found in $PATH
[09:59:12] info devcontainer up
[09:59:12] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[09:59:12] info /Users/runner/work/devpod/devpod/cmd/agent/workspace/up.go:100
[09:59:12] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[09:59:12] info /Users/runner/work/devpod/devpod/cmd/agent/workspace/up.go:57
[09:59:12] info github.com/spf13/cobra.(*Command).execute

for reference, here is my ~/.kube/config file:

apiVersion: v1
clusters:
- cluster:
    server: https://testing20.api.gygkube.com
  name: testing20
contexts:
- context:
    cluster: testing20
    user: arn:aws:eks:eu-central-1:371719169940:cluster/testing20
  name: testing20
current-context: testing20
kind: Config
preferences: {}
users:
- name: alex.eftimie@getyourguide.com
  user:
    auth-provider:
      config:
        client-id: redacted
        client-secret: redacted
        id-token: redacted
        idp-issuer-url: https://accounts.google.com/
        refresh-token: redacted
      name: oidc

- name: arn:aws:eks:eu-central-1:371719169940:cluster/testing20
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      args:
      - eks-login
      - --cluster-name
      - testing20
      command: gygaws
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false
alexef commented 1 year ago

after a few iterations (forcing /usr/local/bin/kubectl, forcing /usr/local/bin/gygaws inside the kubeconfig), I'm still blocked at:

[10:05:02] debug Run command: /usr/local/bin/kubectl --namespace devpod --context testing20 get pvc devpod-dss-kubernetes-default-d3f328346a1-d77b7 --ignore-not-found -o json
[10:05:07] debug error sending telemetry request: Post "https://admin.loft.sh/analytics/v1/devpod/v1/events": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[10:05:10] info error getting credentials - err: exec: "docker-credential-desktop": executable file not found in $PATH, out

something broke w.r.t. $PATH after the 0.3.4 release.

which docker-credential-desktop
/usr/local/bin/docker-credential-desktop

path contains usr/local/bin:

echo $PATH
/Users/alex/Downloads/google-cloud-sdk/bin:/Users/alex/.rd/bin:/Users/alex/.nvm/versions/node/v16.20.0/bin:/Users/alex/.jenv/shims:/Users/alex/.jenv/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/alex/Library/Python/3.9/bin:/Users/alex/.gyg/bin:/Users/alex/.local/bin:/Users/alex/Library/Python/3.10/bin:/Users/alex/Library/Python/3.11/bin:/Users/alex/Library/Python/3.9/bin:/Users/alex/.gyg/bin:/Users/alex/go/bin:/Users/alex/.local/bin:/Users/alex/Library/Python/3.10/bin:/Users/alex/Library/Python/3.11/bin
pascalbreuninger commented 1 year ago

Hi @alexef, this problem only exists when using the UI, right? We've faced some issues regarding the PATH environment variable before, specifically with newlines in it. Could you paste the output of printenv PATH here please?

alexef commented 1 year ago

@pascalbreuninger I only tried the UI so far, I'll also check the CLI.

here's my path:

printenv PATH
/Users/alex/Downloads/google-cloud-sdk/bin:/Users/alex/.rd/bin:/Users/alex/.nvm/versions/node/v16.20.0/bin:/Users/alex/.jenv/shims:/Users/alex/.jenv/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/alex/Library/Python/3.9/bin:/Users/alex/.gyg/bin:/Users/alex/.local/bin:/Users/alex/Library/Python/3.10/bin:/Users/alex/Library/Python/3.11/bin:/Users/alex/Library/Python/3.9/bin:/Users/alex/.gyg/bin:/Users/alex/go/bin:/Users/alex/.local/bin:/Users/alex/Library/Python/3.10/bin:/Users/alex/Library/Python/3.11/bin
pascalbreuninger commented 1 year ago

Thank you. Doesn't look suspicious. Let's try to combine the approaches from 0.3.6 and 0.3.4, will create a PR

alexef commented 1 year ago

fixed in https://github.com/loft-sh/devpod/pull/671

pascalbreuninger commented 1 year ago

@alexef did you check if the fix works on your machine already? here's the latest alpha release with it included: https://github.com/loft-sh/devpod/releases/tag/v0.3.7-alpha.1

you'd have to install it manually though

alexef commented 1 year ago

I did check. the path issues are gone. the kubernetes provider still doesn't work for me, I opened a separate issue

pascalbreuninger commented 1 year ago

okay, thanks