search

kubeflow / notebooks

Kubeflow Notebooks lets you run web-based development environments on your Kubernetes cluster by running them inside Pods.
Apache License 2.0
18 stars 14 forks source link

Notebooks 2.0 // Controller // Update local development guide #41

Open thesuperzapper opened 2 months ago

thesuperzapper commented 2 months ago

Right now, if contributor runs make run from inside the ./workspaces/controller/ folder, it will only run the controller loops, and the Validating webhooks are not used, meaning important validations are not applied.

We should clarify how we expect people to run a local version of the workspace controller in the development guide:

  1. To add/change a feature in the controller:
    • Make the changes in the go code.
    • Update some unit tests for the change and run them in envtest with make test
    • Update some e2e tests for the change and run them in kind with make test-e2e
    • Possibly run a version of the controller locally for a remote cluster (see 3)
  2. To add/change a feature of the webhook:
    • Make the changes in the go code.
    • Update some unit tests for the change and run them in envtest with make test
    • Update some e2e tests for the change and run them in kind with make test-e2e
    • You will need to use make deploy to actually run this on a cluster (probably only on local kind so you can run kind load docker-image ...).
  3. To run a local version of the controller on a real remote cluster:
    • Deploy the current version of the controller on the cluster (to get CRDs/Webhooks)
    • Scale the controller to 0 replicas
    • NOTE: we need to separate out the webhook/controller into separate Pods so they can be scaled separately, right now they are in the same Deployment.
    • Run make run and the controller should reconcile the cluster of your current kubectl context.

There are more complicated methods to improve 2, so that a local version of the webhook can be run on a local kind cluster, so I guess we could provide a make run-kind which patches the ValidatingWebhookConfiguration to use https://host.docker.internal:9443/<webhook-endpoint> and generate the appropreate SSH key pairs, see this upstream issue for more ideas:

thesuperzapper commented 3 weeks ago

I have found a possible way to run a local version of the webhook.

We can do something similar to the external-secrets project and use Tilt, which lets you automatically swap Go binaries into a Pod running on the cluster as you are developing, without needing to rebuild the images.

See their dev-guide for more examples of how to do it: