search

habitat-sh / habitat

Modern applications with built-in automation
https://www.habitat.sh
Apache License 2.0
2.61k stars 315 forks source link

Take service readiness into account #5327

Open christophermaier opened 6 years ago

christophermaier commented 6 years ago

This is related to #5326 but somewhat distinct. If health checks can be incorporated into ongoing service state dissemination, they could also be used to gate when the Supervisor broadcasts that it's actually running that service. In other words, we would only introduce new service rumors for services that are healthy and running. This can be particularly useful for services with long startup times, and by extension, the services that depend on them via binds.

Kubernetes does this with their readiness probes.

If we do this, it's also worth considering if it's worth adding a distinct readiness hook.

mwrock commented 6 years ago

Would/could this also impact rolling updates? I have a use case where it can take severla seconds for a run hook to actually get a service into a properly running state. Currently that means that rolling updates bring everyone down because while the code may be update one at a time, the rolling update does not wait for each member to be available.

Ideally a rolling update should move to the next member (or a member should only be marked updated) once the updated member is healthy.

christophermaier commented 6 years ago

@mwrock Yes! I think this should affect rolling updates, too.

themightychris commented 6 years ago

It would be amazing if habitat's lifecycle could remove the need to have sleep loops in any hooks waiting for a service to come up, they usually end up running similar code to the health check

christophermaier commented 6 years ago

@themightychris Yup, I think that's the end goal here. The current structure of the code makes that a bit hard, but we've got a pathway to fixing that.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.