search

guardian / grid

The Guardian’s image management system
https://www.theguardian.com/info/developer-blog/2015/aug/12/open-sourcing-grid-image-service
Apache License 2.0
1.44k stars 120 forks source link

restart syncchecker on failure with backoff #4131

Closed andrew-nowak closed 1 year ago

andrew-nowak commented 1 year ago

What does this change?

Sometimes Elasticsearch is unavailable, and queries fail with 502. Akka streams by default exit on failures, which is reasonable. We can wrap the source in a RestartSource https://doc.akka.io/docs/akka/current/stream/operators/RestartSource/onFailuresWithBackoff.html to restart when the stream exits with failure.

(We could consider doing the same for the Thrall stream, but that seems more resilient to these transient errors, and is monitored by the healthcheck endpoint, causing a redeploy if down. Regardless, for more consideration and a future change)

prout-bot commented 1 year ago

Seen on usage, image-loader, metadata-editor, thrall, leases, cropper, collections, media-api, kahuna (merged by @andrew-nowak 10 minutes and 53 seconds ago) Please check your changes!

prout-bot commented 1 year ago

Seen on auth (merged by @andrew-nowak 11 minutes ago) Please check your changes!