vmware-tanzu-labs / educates-training-platform

A platform for hosting interactive workshop environments in Kubernetes, or on top of a local container runtime.
https://docs.educates.dev
Apache License 2.0
72 stars 18 forks source link

add ability to only switch to new workshop environment if it is healthy #261

Open djschny opened 8 months ago

djschny commented 8 months ago

Is your feature request related to a problem? Please describe.

Today, if the workshop definition is updated, a new workshop environment is created and requests for new sessions will be handled by the new workshop environment, while pre-existing sessions stay with the old workshop environment. However, there is no health check or similar kind of activity done for this. So if the new workshop environment has a problem fully coming online (whether due to resources, misconfiguration, etc.) the new sessions being handled out either won't load or won't work as intended.

Describe the solution you'd like

The thought was to have a built-in health check for the new environment or a hook/object that could be user specified to have more detailed health inspection logic. The idea new sessions would continue to be provided from the old and presumably "stable" workshop environment until the new one has become "healthy".

Describe alternatives you've considered

I thought about trying to influence or block the workshop environment object, but could not come up with anything that I thought would work.

Additional information

No response

jorgemoralespou commented 7 months ago

We have had a problem where in the setup script we're trying to push code to the "session" git server, but it was not yet available and hence setup failed.

git -C /home/eduk8s/.lab-files/repos/lab push --set-upstream https://git-w47-s001.<REDACTED>/lab main
remote: upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111
fatal: unable to access 'https://git-w47-s001.<REDACTED>/lab/': The requested URL returned error: 503