Open dankingtech opened 7 months ago
Actually, it looks like /v1/conductors/
would be a better endpoint than /v1/nodes/
as it would usually require less work than the later.
To repeat what I mentioned on IRC: any meaningful endpoint will require authentication. So we need to make sure the healtcheck script can use it.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues will close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close
.
/lifecycle stale
/remove-lifecycle stale /triage accepted
With https://github.com/metal3-io-bot/ironic-image/commit/e44c4f731bdfb37e7cd3e2c239c57f43d58c51fb, we now have a path forward. We also need to finish the discussion around https://github.com/metal3-io/baremetal-operator/pull/1685 since it affects how we get the credentials.
/kind feature /help
@dtantsur: This request has been marked as needing help from a contributor.
Please ensure the request meets the requirements listed here.
If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help
command.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues will close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close
.
/lifecycle stale
/remove-lifecycle stale
What steps did you take and what happened:
By any method, cause the connection to the database to fail. Even though any requests to do actual work will fail, the deployment will be seen as live and ready because the health checks are only checking to see that the base URL is responding which it can do even if the internal connections are down. For instance, if one were to attempt to connect to
http://127.0.0.1:6385/v1/nodes/
or other such endpoints, there may be an error but Kubernetes does not know about it.What did you expect to happen:
Ideally, the liveness probe should detect that at least some other endpoint is successful which relies upon the database connection note that the Ironic instance is not healthy.
Anything else you would like to add:
Unfortunately, I have noticed that there are various occasions when Ironic, for various reasons, may fail to be able to connect to the database. In the past I have seen this caused by the database itself having issues as well as other issues related directly to the running instance of the Ironic API. In most cases, simply restarting Ironic has resolved the issue. Regardless, if the backend is unavailable, Ironic serves little utility. Therefore, I recommend changing the
livenessProbe
to check/v1/nodes/
rather than just/
. The same may be true of the Inspector as well, by adding/v1/rules
.Environment:
main
currently./kind bug