search

uselagoon / lagoon

Lagoon, the developer-focused application delivery platform
https://docs.lagoon.sh/
Apache License 2.0
567 stars 148 forks source link

Ability to use startup probes #3527

Open domcyrus opened 1 year ago

domcyrus commented 1 year ago

Startup probes offer a solution for workloads that require substantial initialization before the container is ready. By employing startup probes, we can delay the evaluation of live and ready probes until the startup probe is first confirmed as healthy.

This feature is particularly beneficial for AI workloads, where a large model may need to be initialized or downloaded. Without startup probes, the liveness and/or readiness probe could fail before the initialization completes.

While configuring larger values for ready and live probe checks (#2530) could potentially address this issue, it would be advantageous to make the startup probe configurable through Lagoon, providing a more flexible solution.

domcyrus commented 1 year ago

@Schnitzel FYI

shreddedbacon commented 1 year ago

Is this for user workloads? So someone running their application using Lagoon? Or are you talking about lagoon-core here?

domcyrus commented 1 year ago

It's for use workloads.

derytim commented 1 week ago

We'd love to be able to use startup probes for some services in Lagoon. Any traction here?

shreddedbacon commented 1 week ago

At the moment, there is no plans to progress with support for this. Additionally, if we were to proceed, we would probably still incorporate some limits/restrictions around what can/can't be changed to reduce the potential for pods to take too long to start.

If you have a real use case for this, I'd love to understand more about it. Generally you want your containers to start up quickly, so increasing the starting probe of a pod can have many negative impacts too, especially when it comes to auto-scaling under excessive traffic or general cluster node scaling events. There is a real risk to outages while a pod is starting, because it won't be able to serve requests.

Happy to hear more about what the issue you're facing is, and how startup probes would help you though.