search

kurtosis-tech / kurtosis

A platform for packaging and launching ephemeral backend stacks with a focus on approachability for the average developer.
https://docs.kurtosistech.com/
Apache License 2.0
352 stars 50 forks source link

Enable multiple concurrent kurtosis engines to run in Kubernetes #2065

Open barnabasbusa opened 8 months ago

barnabasbusa commented 8 months ago

Background & motivation

It would be great to allow multiple kurtosis engines to orchestrate different enclaves at the same time given a Kubernetes backend.

Currently if you try to run up more than one kt engines at the same time you will meet with the following error message:

INFO[2024-01-22T07:47:50Z] Starting Kurtosis engine from image 'kurtosistech/engine:0.86.8'... 
Error:  An error occurred starting the Kurtosis engine
  Caused by: An error occurred starting the engine with the engine existence guarantor
  Caused by: An error occurred guaranteeing that a Kurtosis engine is running
  Caused by: We couldn't guarantee that a Kurtosis engine is running because we found a running engine container whose server isn't responding; because this is a strange state, we don't automatically try to correct the problem so you'll want to manually  restart the server by running 'kurtosis engine restart'

This issue arises because the engines don't have dynamic port map-ability.

Desired behaviour

Ideally during a kurtosis run you should be optionally specify what engine and gateway you would like to target.

Optional extra brownie point if you can name the engines and gateways, so you could do something like:

kurtosis run --enclave enclave-gyani --engine engine-gyani --gateway gateway-gyani ....

Ideally the new enclave/engine/gateway would all be created during kurtosis runtime.

This FR will probably enable multi user capabilities in kurtosis cloud too.

How important is this to you?

Painful; the lack of this feature makes using Kurtosis frictionful.

What area of the product does this pertain to?

CLI: the Command Line Interface

h4ck3rk3y commented 8 months ago

So the current architecture blocks the following workflows

  1. If multiple people on the same Kubernetes cluster are using Kurtosis the same time then one person restarting the engine affects other people starting new enclaves even though its parallel
  2. Running GitHub actions against the same cluster is confusing because the thing might clean up more than it should, if we do kurtosis clean -a
  3. People want to run multiple different projects against the same cluster and they want the engine's to be independent - so that long living enclaves don't get accidentally cleaned up
mieubrisse commented 8 months ago

Thought: this sounds like a usecase for "multiple users on a single Kurtosis cluster", which isn't something we support first-class right now.

If we had the concept of users & permissions in a Kurtosis cluster, we could guarantee that a user can only clean up the stuff they're allowed to.

This would also bring us closer to a world I'd like, where Kurtosis is "Notion but for dev/test environments"