ddiss / icyci

Safe and scalable continuous testing, without the bloat
GNU Affero General Public License v3.0
2 stars 1 forks source link

more resilience against network / server outages #9

Open ddiss opened 1 year ago

ddiss commented 1 year ago

The set-and-forget goals of icyCI are currently hindered by a lack of resilience against network and server outages. Most git commands which may contact a remote server already include a retry loop, but the number of retries are hardcoded and there's no back-off. It might be worth dropping the max-retry count altogether and instead just rely on the user-configurable individual state timeouts.

ddiss commented 3 months ago

It might be worth dropping the max-retry count altogether and instead just rely on the user-configurable individual state timeouts.

I think individual state timeouts would be the simplest way to proceed (for implementation and use), but it'd likely make sense to also allow configurable retry back-off times, e.g. via initial delay and back-off exponent.

More effort will need to be put into simulating outages between git client/server, as icyci_test.go currently fetches and pushes everything via local directories. https://git-scm.com/docs/git#Documentation/git.txt-codeGITSSHcode indicates that we might be able to use the GIT_SSH envvar for this.