guardian / grid

The Guardian’s image management system
https://www.theguardian.com/info/developer-blog/2015/aug/12/open-sourcing-grid-image-service
Apache License 2.0
1.44k stars 120 forks source link

recover from file not existing in the S3 IngestQueueBucket #4224

Closed twrichards closed 9 months ago

twrichards commented 9 months ago

Co-authored-by: @dblatcher

Despite the improved permissions in https://github.com/guardian/editorial-tools-platform/pull/744 our logic for handling when the S3 object referenced in the SQS message doesn't exist (albeit rare) is still problematic, the message is retried every 30mins indefinitely because we don't recover from the specific scenario BEFORE deleting the SQS message... this PR along with https://github.com/guardian/editorial-tools-platform/pull/746 puts that right!

github-actions[bot] commented 9 months ago

Deploy build 12234 to TEST

All deployment options - [Deploy build 12234 to TEST](https://riffraff.gutools.co.uk/deployment/deployAgain?project=media-service%3A%3Agrid%3A%3Aall&build=12234&stage=TEST&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 12234 to TEST by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=media-service%3A%3Agrid%3A%3Aall&build=12234&stage=TEST&updateStrategy=MostlyHarmless)

From guardian/actions-riff-raff.

twrichards commented 9 months ago

Tested in TEST ✅ ... by adding a delay to the queue [manually via console] then deleting the file from the ingest bucket before image-loader was able to pick it up, and successfully observed... image

prout-bot commented 9 months ago

Seen on auth, usage, image-loader, metadata-editor, leases, cropper, media-api, kahuna (merged by @twrichards 8 minutes and 45 seconds ago) Please check your changes!

prout-bot commented 9 months ago

Seen on collections (merged by @twrichards 8 minutes and 56 seconds ago) Please check your changes!

prout-bot commented 9 months ago

Seen on thrall (merged by @twrichards 9 minutes and 1 second ago) Please check your changes!