tilt-dev / tilt

Define your dev environment as code. For microservice apps on Kubernetes.
https://tilt.dev/
Apache License 2.0
7.63k stars 301 forks source link

Configuring Tilt to start a new kind cluster isn't working as expected #6446

Closed L30Bola closed 3 hours ago

L30Bola commented 3 hours ago

Expected Behavior

My resources would be deployed as expected.

Current Behavior

My current Tiltfile is as follow:


if config.tilt_subcommand == "up":
    kind_clusters=local("kind get clusters")
    if str(kind_clusters).find("application") == -1:
        local("kind create cluster --config ./kind-config.yaml")
    local("kubectx kind-application")

if config.tilt_subcommand == "down":
    local("kind delete cluster -n application")

allow_k8s_contexts("kind-application")

k8s_yaml("tilt/namespace.yaml")

The kind-config.yaml is as following (although I think it's unnecessary, sending in case someone wants to see it as well):


kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: application
nodes:
- role: control-plane
  image: kindest/node:v1.27.17@sha256:3fd82731af34efe19cd54ea5c25e882985bafa2c9baefe14f8deab1737d9fabe

The namespace.yaml is as following (a simple namespace manifest):


apiVersion: v1
kind: Namespace
metadata:
  name: application

When I execute a tilt up, the cluster is being created as expected but when it reaches the step to create the namespace resource, I receive this error:

image

Build Failed: could not set up kubernetes client: Loading Kubernetes config: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

Steps to Reproduce

  1. Copy the above files
  2. Run tilt up

Context

tilt doctor Output

$ tilt doctor
Tilt: v0.33.20, built 2024-09-03
System: linux-amd64
---
Docker
- Host: unix:///var/run/docker.sock
- Server Version: 27.2.0
- API Version: 1.47
- Builder: 2
- Compose Version: v2.29.2
---
Kubernetes
- Env: kind
- Context: kind-app
- Cluster Name: kind-app
- Namespace: default
- Container Runtime: containerd
- Version: v1.27.16
- Cluster Local Registry: none

About Your Use Case

If I stop tilt and re-run it without running a tilt down (a ctrl+c and then tilt run), everything works as expected. The above error is probably due to the fact that Tilt runs my kubeconfig when it's initializing and when it tries to create a new resource, the kubeconfig that it read when it was starting didn't have the just-created cluster configuration. Is there a way to make Tilt re-read the kubeconfig manually?

Of course, I can split the cluster creation from Tilt, as I've done other times, just this time I wanted to "merge everything" into a simple tilt up.

nicks commented 3 hours ago

Sorry to hear you're having trouble

This is working as expected -- tilt creates one kubernetes connection at startup, and won't switch to a new cluster if you create one at runtime. For more info, see:

https://github.com/tilt-dev/tilt/issues/5266 https://github.com/tilt-dev/tilt/issues/2123