Configure units to automatically restart under certain failure conditions

[wallrj]

We'd like to be able to configure units to automatically restart under certain failure conditions.

• http://www.freedesktop.org/software/systemd/man/systemd.service.html#Restart=

Eg

• If a user deliberately stops the docker container associated with a gear unit.
• Or if the docker container image takes so long to download that the systemd unit times out and enters the failed state. (although that may be an issue that we should address by pre-caching the docker image before calling gear install --start

That last problem may also be addressed by the docker readiness notification idea in this PR https://github.com/openshift/geard/pull/175

The idea of recommending the systemd restart feature by default is described here:

• https://fedorahosted.org/fpc/ticket/191

See:

• Original mailing list discussion at http://lists.openshift.redhat.com/openshift-archives/users/2014-July/msg00026.html
• An example of how this affects us in flocker: https://github.com/ClusterHQ/flocker/issues/326#issuecomment-49453802

[smarterclayton]

Is there a max restart limit that would make sense for all containers (ie something that would be a good default)?

[wallrj]

Sorry for delayed reply.

I can't see a "max restart limit" option. There's http://www.freedesktop.org/software/systemd/man/systemd.service.html#StartLimitInterval=

Supervisord sets a default maximum restart limit of 3 http://supervisord.org/configuration.html#program-x-section-settings

Twisted Procmon doesn't implement a max restart limit; it just allows you to limit the rate with a backoff algorithm. https://twistedmatrix.com/documents/current/api/twisted.runner.procmon.ProcessMonitor.html

Also worth noting that Docker (1.2.0) now has a "restart policy" feature which continues to attempt restarts indefinitely....but with a backoff:

• https://github.com/docker/docker/issues/7226
• https://github.com/docker/docker/pull/7414