search

garden-io / garden

Automation for Kubernetes development and testing. Spin up production-like environments for development, testing, and CI on demand. Use the same configuration and workflows at every step of the process. Speed up your builds and test runs via shared result caching
https://garden.io
Mozilla Public License 2.0
3.33k stars 269 forks source link

k3s and its flavors (k3d and Rancher Desktop) are incompatible with Garden #4528

Closed worldofgeese closed 1 year ago

worldofgeese commented 1 year ago

Bug

This is a tracking issue highlighting the challenges and inconsistent behavior of running Garden with k3s and its family of distributions (k3d and Rancher Desktop). Garden will fail on all flavors waiting for Garden's NGINX Ingress Controller Helm chart to deploy. This is despite Helm showing the chart as ready. In addition, the quickstart-example's URL endpoints such as http://vote.local.demo.garden, are inaccesible.

Current Behavior

Attempt to deploy quickstart-example

```console ❯ garden deploy -l4 Deploy ℹ cloud-dashboard [debug] → Initializing Garden Cloud API client. [debug] No auth token found, proceeding without access to Cloud Dashboard. Command results for this command run will not be available in Cloud Dashboard. [debug] Error reading safe directories from the .gitconfig: Error: Command "git config --get-all safe.directory" failed with code 1: Garden v0.13 (Bonsai) is a major release with significant changes. Please help us improve it by reporting any issues/bugs here: https://go.garden.io/report-bonsai → Run garden util hide-warning 0.13-bonsai to disable this warning. ℹ garden → Running in Garden environment local.vote-demo-quickstart-taohansen [debug] Scanning project root at /home/taohansen/Downloads/projects/quickstart-example-rd → Includes: **/*garden.y*ml → Excludes: .garden/**/*,.git,.gitmodules,.garden/**/*,debug-info*/**,.garden/**/* [debug] Found 7 files in project root /home/taohansen/Downloads/projects/quickstart-example-rd [debug] Scanned and found 18 actions, 0 workflows and 0 modules ℹ providers → Getting status... ℹ providers [verbose] → resolve provider exec is ready. ℹ providers [verbose] → resolve provider templated is ready. ℹ providers [verbose] → resolve provider container is ready. ℹ local-kubernetes [debug] → No kubectl context auto-detected, using first available: rancher-desktop ℹ local-kubernetes [debug] → Found kind with the following version details kind v0.19.0 go1.20.4 linux/amd64 ℹ local-kubernetes [debug] → Kubernetes: Getting API resource info for group apps/v1 ℹ local-kubernetes [debug] → An attempt to get kind daemonset failed with Error: StatusCodeError from Kubernetes API (/apis/apps/v1/namespaces/kube-system/daemonsets/kindnet) - 404 - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"daemonsets.apps \"kindnet\" not found","reason":"NotFound","details":{"name":"kindnet","group":"apps","kind":"daemonsets"},"code":404} ℹ providers [verbose] → Creating namespace vote-demo-quickstart-taohansen ℹ providers [verbose] → Creating namespace garden-system ℹ providers [debug] → Kubernetes: Getting API resource info for group networking.k8s.io/v1 ℹ garden system [debug] → Initializing... ℹ garden system [debug] → Scanning project root at /home/taohansen/.garden/bin/static/kubernetes/system → Includes: **/*garden.y*ml → Excludes: ../../../../../Downloads/projects/quickstart-example-rd/.garden/kubernetes.garden/**/*,.git,.gitmodules,.garden/**/*,debug-info*/**,.garden/**/* ℹ garden system [debug] → Found 6 files in project root /home/taohansen/.garden/bin/static/kubernetes/system ℹ garden system [debug] → Scanned and found 0 actions, 0 workflows and 6 modules ℹ providers → Getting status... ℹ providers [verbose] → resolve provider templated is ready. ℹ providers [verbose] → resolve provider container is ready. ℹ providers [verbose] → resolve provider exec is ready. ℹ local-kubernetes [debug] → Found kind with the following version details kind v0.19.0 go1.20.4 linux/amd64 ℹ local-kubernetes [debug] → An attempt to get kind daemonset failed with Error: StatusCodeError from Kubernetes API (/apis/apps/v1/namespaces/kube-system/daemonsets/kindnet) - 404 - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"daemonsets.apps \"kindnet\" not found","reason":"NotFound","details":{"name":"kindnet","group":"apps","kind":"daemonsets"},"code":404} ℹ providers [verbose] → resolve provider local-kubernetes is ready. ✔ providers → Done (took 0.3 sec) ℹ graph → Resolving actions and modules... ℹ graph [debug] → Scanning module ingress-controller root at /home/taohansen/.garden/bin/static/kubernetes/system/ingress-controller → Includes: *.yaml,*.yml → Excludes: (none) ℹ graph [debug] → Scanning module util root at /home/taohansen/.garden/bin/static/kubernetes/system/util → Includes: *.yaml,*.yml → Excludes: (none) ℹ graph [debug] → Found 1 files in module ingress-controller root /home/taohansen/.garden/bin/static/kubernetes/system/ingress-controller ℹ graph [debug] → Found 3 files in module util root /home/taohansen/.garden/bin/static/kubernetes/system/util ℹ graph [debug] → Converting kubernetes module nginx-ingress-class to actions ℹ graph [debug] → Converting container module default-backend to actions ℹ graph [debug] → Converting kubernetes module nginx-kind-old to actions ℹ graph [debug] → Converting kubernetes module nginx-kind-new to actions ℹ graph [debug] → Converting helm module ingress-controller to actions ℹ graph [debug] → Converting helm module util to actions ℹ graph [debug] → Module nginx-ingress-class converted to 1 actions ℹ graph [debug] → Module default-backend converted to 1 actions ℹ graph [debug] → Module nginx-kind-old converted to 1 actions ℹ graph [debug] → Module nginx-kind-new converted to 1 actions ℹ graph [debug] → Module ingress-controller converted to 2 actions ℹ graph [debug] → Module util converted to 2 actions ℹ graph [debug] → Scanning Deploy action default-backend root at /home/taohansen/.garden/bin/static/kubernetes/system/default-backend → Includes: (none) → Excludes: (none) ℹ graph [debug] → Scanning Deploy action ingress-controller root at /home/taohansen/.garden/bin/static/kubernetes/system/ingress-controller → Includes: (none) → Excludes: (none) ℹ graph [debug] → Scanning Deploy action util root at /home/taohansen/.garden/bin/static/kubernetes/system/util → Includes: *,charts/**/*,templates/**/* → Excludes: (none) ℹ graph [debug] → Found 2 files in Deploy action ingress-controller root /home/taohansen/.garden/bin/static/kubernetes/system/ingress-controller ℹ graph [debug] → Found 7 files in Deploy action util root /home/taohansen/.garden/bin/static/kubernetes/system/util ℹ graph [debug] → Found 2 files in Deploy action default-backend root /home/taohansen/.garden/bin/static/kubernetes/system/default-backend ✔ graph → Done (took 0.5 sec) ℹ deploy.default-backend [verbose] → resolve Deploy type=container name=default-backend (from module default-backend) is ready. ℹ deploy.ingress-controller [verbose] → resolve Deploy type=helm name=ingress-controller (from module ingress-controller) is ready. ℹ deploy.default-backend [debug] → Kubernetes: Getting API resource info for group v1 ℹ deploy.default-backend [verbose] → All resources missing from cluster ✔ garden system [debug] → Done ℹ providers → Preparing environment... ℹ local-kubernetes → Configuring... ℹ garden system [debug] → Initializing... ℹ providers → Getting status... ℹ providers [verbose] → resolve provider container is ready. ℹ local-kubernetes [debug] → Found kind with the following version details kind v0.19.0 go1.20.4 linux/amd64 ℹ local-kubernetes [debug] → An attempt to get kind daemonset failed with Error: StatusCodeError from Kubernetes API (/apis/apps/v1/namespaces/kube-system/daemonsets/kindnet) - 404 - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"daemonsets.apps \"kindnet\" not found","reason":"NotFound","details":{"name":"kindnet","group":"apps","kind":"daemonsets"},"code":404} ℹ providers [verbose] → resolve provider local-kubernetes is ready. ✔ providers → Cached (took 0.3 sec) ℹ providers → Run with --force-refresh to force a refresh of provider statuses. ℹ garden system [debug] → Scanning project root at /home/taohansen/.garden/bin/static/kubernetes/system → Includes: **/*garden.y*ml → Excludes: ../../../../../Downloads/projects/quickstart-example-rd/.garden/kubernetes.garden/**/*,.git,.gitmodules,.garden/**/*,debug-info*/**,.garden/**/* ℹ garden system [debug] → Found 6 files in project root /home/taohansen/.garden/bin/static/kubernetes/system ℹ garden system [debug] → Scanned and found 0 actions, 0 workflows and 6 modules ℹ providers → Getting status... ℹ providers [verbose] → resolve provider container is ready. ℹ providers [verbose] → resolve provider local-kubernetes is ready. ℹ providers [verbose] → resolve provider templated is ready. ℹ providers [verbose] → resolve provider exec is ready. ✔ providers → Cached (took 0 sec) ℹ providers → Run with --force-refresh to force a refresh of provider statuses. ℹ graph → Resolving actions and modules... ℹ graph [debug] → Scanning module ingress-controller root at /home/taohansen/.garden/bin/static/kubernetes/system/ingress-controller → Includes: *.yaml,*.yml → Excludes: (none) ℹ graph [debug] → Scanning module util root at /home/taohansen/.garden/bin/static/kubernetes/system/util → Includes: *.yaml,*.yml → Excludes: (none) ℹ graph [debug] → Found 1 files in module ingress-controller root /home/taohansen/.garden/bin/static/kubernetes/system/ingress-controller ℹ graph [debug] → Found 3 files in module util root /home/taohansen/.garden/bin/static/kubernetes/system/util ℹ graph [debug] → Converting kubernetes module nginx-kind-old to actions ℹ graph [debug] → Converting kubernetes module nginx-ingress-class to actions ℹ graph [debug] → Converting kubernetes module nginx-kind-new to actions ℹ graph [debug] → Converting container module default-backend to actions ℹ graph [debug] → Converting helm module ingress-controller to actions ℹ graph [debug] → Converting helm module util to actions ℹ graph [debug] → Module nginx-kind-old converted to 1 actions ℹ graph [debug] → Module nginx-ingress-class converted to 1 actions ℹ graph [debug] → Module nginx-kind-new converted to 1 actions ℹ graph [debug] → Module default-backend converted to 1 actions ℹ graph [debug] → Module ingress-controller converted to 2 actions ℹ graph [debug] → Module util converted to 2 actions ℹ graph [debug] → Scanning Deploy action default-backend root at /home/taohansen/.garden/bin/static/kubernetes/system/default-backend → Includes: (none) → Excludes: (none) ℹ graph [debug] → Scanning Deploy action ingress-controller root at /home/taohansen/.garden/bin/static/kubernetes/system/ingress-controller → Includes: (none) → Excludes: (none) ℹ graph [debug] → Scanning Deploy action util root at /home/taohansen/.garden/bin/static/kubernetes/system/util → Includes: *,charts/**/*,templates/**/* → Excludes: (none) ℹ graph [debug] → Found 2 files in Deploy action ingress-controller root /home/taohansen/.garden/bin/static/kubernetes/system/ingress-controller ℹ graph [debug] → Found 7 files in Deploy action util root /home/taohansen/.garden/bin/static/kubernetes/system/util ℹ graph [debug] → Found 2 files in Deploy action default-backend root /home/taohansen/.garden/bin/static/kubernetes/system/default-backend ✔ graph → Done (took 0.5 sec) ℹ deploy.default-backend [verbose] → resolve Deploy type=container name=default-backend (from module default-backend) is ready. ℹ deploy.ingress-controller [verbose] → resolve Deploy type=helm name=ingress-controller (from module ingress-controller) is ready. ℹ deploy.default-backend [verbose] → All resources missing from cluster ℹ deploy.default-backend → missing ℹ deploy.default-backend → Deploying version v-2123cd3582... ℹ deploy.default-backend [verbose] → All resources missing from cluster ℹ deploy.ingress-controller → missing ℹ deploy.default-backend [debug] → Kubernetes: Getting API resource info for group batch/v1 ℹ deploy.default-backend [debug] → Kubernetes: Getting API resource info for group batch/v1beta1 ℹ deploy.default-backend [debug] → Kubernetes: Getting API resource info for group extensions/v1beta1 ℹ deploy.default-backend → Waiting for resources to be ready... ℹ deploy.default-backend [verbose] → [kubernetes-plugin] Waiting for resources to be ready... ℹ deploy.default-backend [debug] → Status of Deployment default-backend is "ready" ℹ deploy.default-backend [verbose] → [kubernetes-plugin] Status of Deployment default-backend is "ready" ℹ deploy.default-backend [debug] → Status of Service default-backend is "ready" ℹ deploy.default-backend [verbose] → [kubernetes-plugin] Status of Service default-backend is "ready" ℹ deploy.default-backend [verbose] → [kubernetes-plugin] Resources ready ℹ deploy.default-backend → Resources ready ℹ deploy.default-backend [debug] → Getting currently deployed resource statuses... ℹ deploy.default-backend [debug] → Comparing expected and deployed resources... ℹ deploy.default-backend [debug] → All resources match. ✔ deploy.default-backend → Done (took 2.8 sec) ℹ deploy.default-backend [verbose] → Deploy type=container name=default-backend (from module default-backend) is ready. ℹ deploy.ingress-controller → Deploying version v-866b0ce415... ℹ deploy.ingress-controller [verbose] → [helm] NAME: garden-nginx ℹ deploy.ingress-controller [verbose] → [helm] LAST DEPLOYED: Sat Jun 3 17:51:27 2023 ℹ deploy.ingress-controller [verbose] → [helm] NAMESPACE: garden-system ℹ deploy.ingress-controller [verbose] → [helm] STATUS: deployed ℹ deploy.ingress-controller [verbose] → [helm] REVISION: 1 ℹ deploy.ingress-controller [verbose] → [helm] TEST SUITE: None ℹ deploy.ingress-controller [verbose] → [helm] NOTES: ℹ deploy.ingress-controller [verbose] → [helm] The ingress-nginx controller has been installed. ℹ deploy.ingress-controller [verbose] → [helm] It may take a few minutes for the LoadBalancer IP to be available. ℹ deploy.ingress-controller [verbose] → [helm] You can watch the status by running 'kubectl --namespace garden-system get services -o wide -w garden-nginx-ingress-nginx-controller' ℹ deploy.ingress-controller [verbose] ℹ deploy.ingress-controller [verbose] → [helm] An example Ingress that makes use of the controller: ℹ deploy.ingress-controller [verbose] → [helm] apiVersion: networking.k8s.io/v1 ℹ deploy.ingress-controller [verbose] → [helm] kind: Ingress ℹ deploy.ingress-controller [verbose] → [helm] metadata: ℹ deploy.ingress-controller [verbose] → [helm] name: example ℹ deploy.ingress-controller [verbose] → [helm] namespace: foo ℹ deploy.ingress-controller [verbose] → [helm] spec: ℹ deploy.ingress-controller [verbose] → [helm] ingressClassName: nginx ℹ deploy.ingress-controller [verbose] → [helm] rules: ℹ deploy.ingress-controller [verbose] → [helm] - host: www.example.com ℹ deploy.ingress-controller [verbose] → [helm] http: ℹ deploy.ingress-controller [verbose] → [helm] paths: ℹ deploy.ingress-controller [verbose] → [helm] - backend: ℹ deploy.ingress-controller [verbose] → [helm] service: ℹ deploy.ingress-controller [verbose] → [helm] name: exampleService ℹ deploy.ingress-controller [verbose] → [helm] port: ℹ deploy.ingress-controller [verbose] → [helm] number: 80 ℹ deploy.ingress-controller [verbose] → [helm] path: / ℹ deploy.ingress-controller [verbose] → [helm] # This section is only required if TLS is to be enabled for the Ingress ℹ deploy.ingress-controller [verbose] → [helm] tls: ℹ deploy.ingress-controller [verbose] → [helm] - hosts: ℹ deploy.ingress-controller [verbose] → [helm] - www.example.com ℹ deploy.ingress-controller [verbose] → [helm] secretName: example-tls ℹ deploy.ingress-controller [verbose] ℹ deploy.ingress-controller [verbose] → [helm] If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided: ℹ deploy.ingress-controller [verbose] ℹ deploy.ingress-controller [verbose] → [helm] apiVersion: v1 ℹ deploy.ingress-controller [verbose] → [helm] kind: Secret ℹ deploy.ingress-controller [verbose] → [helm] metadata: ℹ deploy.ingress-controller [verbose] → [helm] name: example-tls ℹ deploy.ingress-controller [verbose] → [helm] namespace: foo ℹ deploy.ingress-controller [verbose] → [helm] data: ℹ deploy.ingress-controller [verbose] → [helm] tls.crt: ℹ deploy.ingress-controller [verbose] → [helm] tls.key: ℹ deploy.ingress-controller [verbose] → [helm] type: kubernetes.io/tls ℹ deploy.ingress-controller → Waiting for resources to be ready... ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Waiting for resources to be ready... ℹ deploy.ingress-controller [debug] → Kubernetes: Getting API resource info for group rbac.authorization.k8s.io/v1 ℹ deploy.ingress-controller [debug] → Status of ServiceAccount garden-nginx-ingress-nginx is "ready" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of ServiceAccount garden-nginx-ingress-nginx is "ready" ℹ deploy.ingress-controller [debug] → Status of ConfigMap garden-nginx-ingress-nginx-controller is "ready" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of ConfigMap garden-nginx-ingress-nginx-controller is "ready" ℹ deploy.ingress-controller [debug] → Status of ClusterRole garden-nginx-ingress-nginx is "ready" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of ClusterRole garden-nginx-ingress-nginx is "ready" ℹ deploy.ingress-controller [debug] → Status of ClusterRoleBinding garden-nginx-ingress-nginx is "ready" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of ClusterRoleBinding garden-nginx-ingress-nginx is "ready" ℹ deploy.ingress-controller [debug] → Status of Role garden-nginx-ingress-nginx is "ready" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Role garden-nginx-ingress-nginx is "ready" ℹ deploy.ingress-controller [debug] → Status of RoleBinding garden-nginx-ingress-nginx is "ready" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of RoleBinding garden-nginx-ingress-nginx is "ready" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of DaemonSet garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of DaemonSet garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller → DaemonSet/garden-nginx-ingress-nginx-controller: RELOAD NGINX reload triggered due to a change in configuration ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] DaemonSet/garden-nginx-ingress-nginx-controller: RELOAD NGINX reload triggered due to a change in configuration ℹ deploy.ingress-controller [debug] → Status of IngressClass nginx is "ready" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of IngressClass nginx is "ready" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of DaemonSet garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of DaemonSet garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of DaemonSet garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of DaemonSet garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of DaemonSet garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of DaemonSet garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of DaemonSet garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of DaemonSet garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of DaemonSet garden-nginx-ingress-nginx-controller is "ready" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of DaemonSet garden-nginx-ingress-nginx-controller is "ready" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [debug] → Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Status of Service garden-nginx-ingress-nginx-controller is "deploying" ℹ deploy.ingress-controller [verbose] → [kubernetes-plugin] Timed out waiting for deploy.ingress-controller to deploy after 300 seconds ✖ deploy.ingress-controller → Failed (took 316.3 sec) ✖ deploy.ingress-controller → Failed processing Deploy type=helm name=ingress-controller (from module ingress-controller) (took 319.27 sec). Here is the output: ──────────────────────────────────────────────────────────────────────────────── Timed out waiting for deploy.ingress-controller to deploy after 300 seconds ──────────────────────────────────────────────────────────────────────────────── ✖ providers → Failed processing resolve provider local-kubernetes (took 323.44 sec). Here is the output: ──────────────────────────────────────────────────────────────────────────────── local-kubernetes — an error occurred when configuring environment: Error: Failed to complete 1/2 tasks: ↳ Deploy type=helm name=ingress-controller (from module ingress-controller): Deploy type=helm name=ingress-controller (from module ingress-controller) failed: Error: Timed out waiting for deploy.ingress-controller to deploy after 300 seconds ──────────────────────────────────────────────────────────────────────────────── [debug] Done flushing all events and log entries. Failed resolving one or more providers: - local-kubernetes ```

Don't be fooled, the ingress controller is in fact deployed

❯ helm list -A
NAME            NAMESPACE       REVISION    UPDATED                                     STATUS      CHART                   APP VERSION
garden-nginx    garden-system   1           2023-06-03 17:05:27.968979041 +0200 CEST    deployed    ingress-nginx-4.0.13    1.1.0
```sh ❯ helm status garden-nginx -n garden-system NAME: garden-nginx LAST DEPLOYED: Sat Jun 3 17:05:27 2023 NAMESPACE: garden-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The ingress-nginx controller has been installed. It may take a few minutes for the LoadBalancer IP to be available. You can watch the status by running 'kubectl --namespace garden-system get services -o wide -w garden-nginx-ingress-nginx-controller' An example Ingress that makes use of the controller: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example namespace: foo spec: ingressClassName: nginx rules: - host: www.example.com http: paths: - backend: service: name: exampleService port: number: 80 path: / # This section is only required if TLS is to be enabled for the Ingress tls: - hosts: - www.example.com secretName: example-tls If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided: apiVersion: v1 kind: Secret metadata: name: example-tls namespace: foo data: tls.crt: tls.key: type: kubernetes.io/tls ``` 
❯ kubectl get po -A
NAMESPACE       NAME                                          READY   STATUS    RESTARTS        AGE
kube-system     local-path-provisioner-957fdf8bc-7b5rq        1/1     Running   8 (177m ago)    28h
kube-system     coredns-77ccd57875-sgcbq                      1/1     Running   3 (177m ago)    28h
kube-system     metrics-server-54dc485875-9lfvh               1/1     Running   21 (177m ago)   28h
garden-system   default-backend-55d888f6bc-pbg9x              1/1     Running   0               5m33s
garden-system   garden-nginx-ingress-nginx-controller-rk897   1/1     Running   0               5m27s

The rest of the stack can be deployed with a second garden deploy action.

```console ℹ garden → Running in Garden environment local.vote-demo-quickstart-taohansen ℹ providers → Getting status... ℹ providers → Getting status... ✔ providers → Done (took 0.3 sec) ℹ graph → Resolving actions and modules... ✔ graph → Done (took 0.6 sec) ✔ providers → Done (took 3.6 sec) ℹ graph → Resolving actions and modules... ✔ graph → Done (took 0.6 sec) ℹ deploy.e2e-runner → missing ℹ deploy.postgres → missing ℹ deploy.worker → missing ℹ deploy.redis → missing ℹ deploy.redis → Deploying version v-cf1ad5f6ff... ℹ deploy.postgres → Deploying version v-a3a5f73edd... ℹ build.worker → Already built ℹ deploy.result → missing ℹ deploy.vote → missing ℹ deploy.api → missing ℹ build.api-build → Already built ℹ build.vote → Already built ℹ build.result → Already built ℹ deploy.redis → Waiting for resources to be ready... ℹ deploy.postgres → Waiting for resources to be ready... ℹ deploy.redis → Deployment/redis: Started container redis ℹ deploy.postgres → Deployment/postgres: Started container postgres ℹ deploy.redis → Resources ready ✔ deploy.redis → Done (took 6.7 sec) ℹ deploy.api → Deploying version v-5d66152406... ⚠ deploy.postgres → Deployment/postgres: Readiness probe failed: psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? ℹ deploy.api → Waiting for resources to be ready... ℹ deploy.api → Deployment/api: Started container api ℹ deploy.postgres → Resources ready ✔ deploy.postgres → Done (took 10.4 sec) ℹ run.db-init → Running... ℹ deploy.api → Resources ready ✔ deploy.api → Done (took 5.8 sec) ℹ deploy.api → Ingress: http://api.local.demo.garden ✔ run.db-init → Done (took 19.1 sec) ℹ deploy.result → Deploying version v-3b58362b03... ℹ deploy.worker → Deploying version v-bd80114b02... ℹ deploy.worker → Waiting for resources to be ready... ℹ deploy.result → Waiting for resources to be ready... ℹ deploy.worker → Deployment/worker: Started container worker ℹ deploy.result → Deployment/result: Started container result ℹ deploy.worker → Resources ready ✔ deploy.worker → Done (took 5.9 sec) ℹ deploy.result → Resources ready ✔ deploy.result → Done (took 6.9 sec) ℹ deploy.result → Ingress: http://result.local.demo.garden ℹ deploy.vote → Deploying version v-29effbce82... ℹ deploy.vote → Waiting for resources to be ready... ℹ deploy.vote → Deployment/vote: Started container vote ⚠ deploy.vote → Deployment/vote: Readiness probe failed: Get "http://10.42.0.121:8080/": dial tcp 10.42.0.121:8080: connect: connection refused ⚠ deploy.vote → Deployment/vote: Readiness probe failed: Get "http://10.42.0.121:8080/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) ℹ deploy.vote → Resources ready ✔ deploy.vote → Done (took 21.4 sec) ℹ deploy.vote → Ingress: http://vote.local.demo.garden ℹ deploy.e2e-runner → Deploying version v-704c2e16d1... ℹ deploy.e2e-runner → Waiting for resources to be ready... ℹ deploy.e2e-runner → Resources ready ✔ deploy.e2e-runner → Done (took 2.7 sec) ```

Endpoints are inaccessible

❯ curl http://vote.local.demo.garden
curl: (7) Failed to connect to vote.local.demo.garden port 80 after 3 ms: Couldn't connect to server

Thanks to @stefreak for pairing with me on this discovery process!

Expected behavior

Garden should deploy the entire quickstart-example in one-shot and expose accessible URL endpoints to the user.

Reproducible example

Clone the quickstart-example to a local project directory and change directory into it. Now set up a k3s distribution of your choice. Instructions for each are provided below.

k3s

curl -sfL https://get.k3s.io | sh -s - --docker --disable=traefik --write-kubeconfig-mode=644
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

k3d

❯ k3d registry create myregistry.localhost --port 12345
INFO[0000] Creating node 'k3d-myregistry.localhost'     
INFO[0000] Successfully created registry 'k3d-myregistry.localhost' 
INFO[0000] Starting Node 'k3d-myregistry.localhost'     
INFO[0000] Successfully created registry 'k3d-myregistry.localhost' 
# You can now use the registry like this (example):
# 1. create a new cluster that uses this registry
k3d cluster create --registry-use k3d-myregistry.localhost:12345

# 2. tag an existing local image to be pushed to the registry
docker tag nginx:latest k3d-myregistry.localhost:12345/mynginx:v0.1

# 3. push that image to the registry
docker push k3d-myregistry.localhost:12345/mynginx:v0.1

# 4. run a pod that uses this image
kubectl run mynginx --image k3d-myregistry.localhost:12345/mynginx:v0.1

Tell our system how to reach the registry

echo "127.0.0.1 k3d-myregistry.localhost" | sudo tee -a /etc/hosts

Then create your cluster

k3d cluster create dev \
  --agents 1 \
  --k3s-arg "--disable=traefik@server:0" \
  --k3s-arg "--disable=servicelb@server:0" \
  --no-lb \
  --registry-use k3d-myregistry.localhost:12345 \
  --wait

Then append the deploymentRegistry declaration

sed -i "/defaultHostname: \${var.base-hostname}/a\    deploymentRegistry:\n      hostname: k3d-myregistry.localhost\n      port: 12345\n      insecure: true\n      namespace: \${kebabCase(local.username)}" project.garden.yml

This will result in the following errors on re-deploy, yet unsolved

```console ❯ garden deploy -l4 Deploy ℹ cloud-dashboard [debug] → Initializing Garden Cloud API client. [debug] No auth token found, proceeding without access to Cloud Dashboard. Command results for this command run will not be available in Cloud Dashboard. [debug] Error reading safe directories from the .gitconfig: Error: Command "git config --get-all safe.directory" failed with code 1: Garden v0.13 (Bonsai) is a major release with significant changes. Please help us improve it by reporting any issues/bugs here: https://go.garden.io/report-bonsai → Run garden util hide-warning 0.13-bonsai to disable this warning. ℹ garden → Running in Garden environment local.vote-demo-quickstart-taohansen [debug] Scanning project root at /home/taohansen/quickstart-example-k3d → Includes: **/*garden.y*ml → Excludes: .garden/**/*,.git,.gitmodules,.garden/**/*,debug-info*/**,.garden/**/* [debug] Found 7 files in project root /home/taohansen/quickstart-example-k3d [debug] Scanned and found 18 actions, 0 workflows and 0 modules ℹ providers → Getting status... ℹ providers [verbose] → resolve provider exec is ready. ℹ providers [verbose] → resolve provider templated is ready. ℹ providers [verbose] → resolve provider container is ready. ℹ local-kubernetes [debug] → Found kind with the following version details kind v0.19.0 go1.20.4 linux/amd64 ℹ local-kubernetes [debug] → Kubernetes: Getting API resource info for group apps/v1 ℹ local-kubernetes [debug] → An attempt to get kind daemonset failed with Error: StatusCodeError from Kubernetes API (/apis/apps/v1/namespaces/kube-system/daemonsets/kindnet) - 404 - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"daemonsets.apps \"kindnet\" not found","reason":"NotFound","details":{"name":"kindnet","group":"apps","kind":"daemonsets"},"code":404} ℹ providers [verbose] → resolve provider local-kubernetes is ready. ✔ providers → Cached (took 1.8 sec) ℹ providers → Run with --force-refresh to force a refresh of provider statuses. ℹ graph → Resolving actions and modules... ℹ graph [debug] → Scanning Build action api-build root at /home/taohansen/quickstart-example-k3d/api → Includes: (none) → Excludes: (none) ℹ graph [debug] → Scanning Build action result root at /home/taohansen/quickstart-example-k3d/result → Includes: (none) → Excludes: (none) ℹ graph [debug] → Scanning Build action vote root at /home/taohansen/quickstart-example-k3d/vote → Includes: . → Excludes: (none) ℹ graph [debug] → Scanning Build action e2e-runner root at /home/taohansen/quickstart-example-k3d/vote → Includes: (none) → Excludes: (none) ℹ graph [debug] → Scanning Build action worker root at /home/taohansen/quickstart-example-k3d/worker → Includes: (none) → Excludes: (none) ℹ graph [debug] → Scanning Deploy action postgres root at /home/taohansen/quickstart-example-k3d/postgres → Includes: (none) → Excludes: (none) ℹ graph [debug] → Scanning Deploy action redis root at /home/taohansen/quickstart-example-k3d/redis → Includes: (none) → Excludes: (none) ℹ graph [debug] → Found 5 files in Build action api-build root /home/taohansen/quickstart-example-k3d/api ℹ graph [debug] → Found 6 files in Build action result root /home/taohansen/quickstart-example-k3d/result ℹ graph [debug] → Found 35 files in Build action vote root /home/taohansen/quickstart-example-k3d/vote ℹ graph [debug] → Found 35 files in Build action e2e-runner root /home/taohansen/quickstart-example-k3d/vote ℹ graph [debug] → Found 4 files in Build action worker root /home/taohansen/quickstart-example-k3d/worker ℹ graph [debug] → Found 1 files in Deploy action postgres root /home/taohansen/quickstart-example-k3d/postgres ℹ graph [debug] → Found 1 files in Deploy action redis root /home/taohansen/quickstart-example-k3d/redis ✔ graph → Done (took 0.9 sec) ℹ deploy.redis [verbose] → resolve Deploy type=container name=redis is ready. ℹ deploy.postgres [verbose] → resolve Deploy type=container name=postgres is ready. ℹ build.api-build [verbose] → resolve Build type=container name=api-build is ready. ℹ build.result [verbose] → resolve Build type=container name=result is ready. ℹ build.vote [verbose] → resolve Build type=container name=vote is ready. ℹ build.worker [verbose] → resolve Build type=container name=worker is ready. ℹ deploy.api [verbose] → resolve Deploy type=container name=api is ready. ℹ run.db-init [verbose] → resolve Run type=container name=db-init is ready. ℹ deploy.result [verbose] → resolve Deploy type=container name=result is ready. ℹ deploy.worker [verbose] → resolve Deploy type=container name=worker is ready. ℹ deploy.vote [verbose] → resolve Deploy type=container name=vote is ready. ℹ deploy.api [debug] → Kubernetes: Getting API resource info for group networking.k8s.io/v1 ℹ deploy.postgres [debug] → Kubernetes: Getting API resource info for group v1 ℹ deploy.e2e-runner [verbose] → resolve Deploy type=container name=e2e-runner is ready. ℹ deploy.e2e-runner [verbose] → All resources missing from cluster ℹ deploy.worker [verbose] → All resources missing from cluster ℹ deploy.e2e-runner → missing ℹ deploy.worker → missing ℹ run.db-init [verbose] → Checking status... ℹ build.worker [debug] → Spawning '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker manifest inspect k3d-myregistry.localhost:12345/taohansen/worker:v-5c1154e6d7' in /home/taohansen/quickstart-example-k3d/.garden/build/worker ✖ build.worker → Failed resolving status for Build type=container name=worker (took 0.09 sec). Here is the output: ──────────────────────────────────────────────────────────────────────────────── Unable to run docker command: An error occurred while trying to run '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker' (spawn /home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker ENOENT). Please make sure '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker' is installed and in the $PATH. ──────────────────────────────────────────────────────────────────────────────── ℹ deploy.vote [verbose] → All resources missing from cluster ℹ deploy.api [verbose] → All resources missing from cluster ℹ deploy.redis [debug] → Getting currently deployed resource statuses... ℹ deploy.result [verbose] → All resources missing from cluster ℹ deploy.postgres [debug] → Getting currently deployed resource statuses... ℹ run.db-init [verbose] → Status check complete ℹ deploy.vote → missing ℹ deploy.api → missing ℹ deploy.result → missing ℹ build.api-build [debug] → Spawning '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker manifest inspect k3d-myregistry.localhost:12345/taohansen/api-build:v-6d254cdf3a' in /home/taohansen/quickstart-example-k3d/.garden/build/api-build ✖ build.api-build → Failed resolving status for Build type=container name=api-build (took 0.17 sec). Here is the output: ──────────────────────────────────────────────────────────────────────────────── Unable to run docker command: An error occurred while trying to run '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker' (spawn /home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker ENOENT). Please make sure '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker' is installed and in the $PATH. ──────────────────────────────────────────────────────────────────────────────── ℹ build.result [debug] → Spawning '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker manifest inspect k3d-myregistry.localhost:12345/taohansen/result:v-ac4c0a14af' in /home/taohansen/quickstart-example-k3d/.garden/build/result ✖ build.result → Failed resolving status for Build type=container name=result (took 0.36 sec). Here is the output: ──────────────────────────────────────────────────────────────────────────────── Unable to run docker command: An error occurred while trying to run '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker' (spawn /home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker ENOENT). Please make sure '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker' is installed and in the $PATH. ──────────────────────────────────────────────────────────────────────────────── ℹ build.vote [debug] → Spawning '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker manifest inspect k3d-myregistry.localhost:12345/taohansen/vote:v-e12f006685' in /home/taohansen/quickstart-example-k3d/.garden/build/vote ✖ build.vote → Failed resolving status for Build type=container name=vote (took 0.53 sec). Here is the output: ──────────────────────────────────────────────────────────────────────────────── Unable to run docker command: An error occurred while trying to run '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker' (spawn /home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker ENOENT). Please make sure '/home/taohansen/.garden/tools/docker/e9744f1ea7f84408/docker/docker' is installed and in the $PATH. ──────────────────────────────────────────────────────────────────────────────── ℹ deploy.redis [debug] → Comparing expected and deployed resources... ℹ deploy.redis [debug] → All resources match. ℹ deploy.redis → Already deployed ℹ deploy.redis [verbose] → Deploy type=container name=redis status is ready. ℹ deploy.postgres [debug] → Comparing expected and deployed resources... ℹ deploy.postgres [debug] → All resources match. ℹ deploy.postgres → Already deployed ℹ deploy.postgres [verbose] → Deploy type=container name=postgres status is ready. [debug] Done flushing all events and log entries. [debug] Watcher: Initializing [debug] Watcher: Starting FSWatcher 5 deploy action(s) failed! ```

Rancher Desktop

For Rancher Desktop supplement the official guide with this guide.

Workaround

A user can run garden deploy a second time to ensure the full stack is deployed. To work around the URL one can copy my Cloud Shell tutorial setup script from line 32 that uses node ports and linkURL rewrites in lieu of an ingress controller.

Additional context

This is fairly important to the kind of user we're targeting. We have customers, community members, reviewers like Viktor Farcic, and the broader Kubernetes enthusiast community who all prefer to use the lightest (CPU and memory sparing) flavor of freely-licensed local Kubernetes for development. Rancher Desktop provides a user-friendly UI to k3s that is growing ever more popular among users. Viktor describes missing support for Rancher Desktop as "very, very, very disappointing"😿

Supporting this family of local Kubernetes is a community win because it meets our users where they're already using Kubernetes and broadens our community base whose first Kubernetes is likely to be local.

Your environment

garden version 0.13.0

Tagging other interested parties @edvald @jalper @ShankyJS

worldofgeese commented 1 year ago

Related to https://github.com/garden-io/garden/issues/4460