Closed Jetserver closed 5 years ago
This is a critical feature for a robust backup solution. Which I want Xen Orchestra to be for our organization.
@julien-f I raised the severity of this. IDK if it's very complicated or not, but it should be at least planned in some future release :+1:
It's getting better but we are not there yet.
@julien-f @olivierlambert how is this coming along?
Any progress? I'm a little surprised that automatic reconnection doesn't exist yet -- it means that Xen Orchestra breaks for our users and requires manual intervention by an admin every few days...
This is looking like a critical requirement for us too. We are too frequently having to manually reconnect pools at remote locations. Makes it hard to sell to other areas of our business as a stable platform upon which to host production services.
I think it exists but it's not optimal, the main difficulty is that there isn't any real "connected" status to the XAPI. It's just a bunch of events (coming from XAPI) so we can't really guess when it's interrupted. A way to do so is to have a timeout if we don't get events from the last minute or so. But this leads to non-trivial consequences, @julien-f is more able to explain this than me (but I wanted to show it's not "trivial")
The /settings/servers
page seems to know whether a host is connected, at least -- when we lose a connection to a host, 'Connected' on that page changes to 'Disconnected'. xo-cli also seems to know whether a server is connected.
Maybe all I need is a cron job which runs xo-cli server.getAll
and runs xo-cli server.connect ...
on each disconnected server.
(Though occasionally XO fails to reconnect to a disconnected server until I restart XO, which is probably a different bug... That seems not to have happened recently though, so maybe that bug has been fixed already :-) )
This should be completely fixed by 485b8fe99
Context
Expected behavior
Try to automatically re-connect a physical host, send a notification alert if connect fails after X times. Wait X hours between each retries.
Current behavior
When dealing with 100+ hosts, in several geo locations through one XO, we sometimes encounter communication issues causing the physical hosts to disconnect.
XO will not try to automatically reconnect, causing the host to stay off line until human interaction.
This is could cause serious issues if there are jobs configured on that host, the jobs will not apply to the host. Sometimes you will discover this when it's too late... (no vm backups for specific host).