Support eventual consistency

[windsource]

Description

When applying a new manifest, currently Ankaios has a fixed number of retries when the start fails and after that finally gives up. The workload remains in state Pending, subState StartingFailed. There can be different reasons why the start fails like

• cannot pull image (registry not available, image not found, not authorized, ...)
• invalid options passed to either commandOptions or commandArgs
• Folder not existing when mouting volumes
• etc.

While some of the problems cannot be solved without changing the manifest (e.g. invalid options) others might disappear after some time (e.g. registry not available or folder not existing).

Some users expect that Ankaios constantly tries to reach the desired that and also that Ankaios provides the result of the latest try (e.g. Podman error message).

Goals

• Ankaios should constantly try to reach the desired state.
• The interval between the retries shall be increased over time with a jitter (maybe same strategy as K8S).
• ank get workloads shall provide the latest result.
• ank apply shall return after the first attempt

Final result

Summary

To be filled when the final solution is sketched.

Tasks

• [ ] Task 1
• [ ] Task 2
• [ ] ...

[windsource]

Maybe we can also have an optional maximum time before Ankaios stops to reach desired state. The parameter could be part of a config file (see #302).

[inf17101]

Builds upon #67 (PR #137)