search

guardian / grid

The Guardian’s image management system
https://www.theguardian.com/info/developer-blog/2015/aug/12/open-sourcing-grid-image-service
Apache License 2.0
1.44k stars 119 forks source link

s3Watcher sends % of files to lower environment Ingest Queue Bucket #4205

Closed dblatcher closed 7 months ago

dblatcher commented 7 months ago

What does this change?

On PROD, S3watcher currently sends random sample of 1% of images from FTP to the ingest bucket for the lower environment (TEST) so there is a stream of incoming files so we can see the lower environment is working. See https://github.com/guardian/grid/pull/4033.

This PR makes S3 watcher a further random sample (size configurable using env var) to the lower environment's Ingest Queue Bucket to test the new infastructure introduced in https://github.com/guardian/grid/pull/4201

## How should a reviewer test this change?

How can success be measured?

When deployed (after the corresponding platform change on this PR), the TEST environment will get a regular stream of images into the new queue mechanism.

The size of the sample will be configurable by manually changing the environment for s3 watcher lambda.

Who should look at this?

Tested? Documented?

github-actions[bot] commented 7 months ago

Deploy build 12154 to TEST

All deployment options - [Deploy build 12154 to TEST](https://riffraff.gutools.co.uk/deployment/deployAgain?project=media-service%3A%3Agrid%3A%3Aall&build=12154&stage=TEST&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 12154 to TEST by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=media-service%3A%3Agrid%3A%3Aall&build=12154&stage=TEST&updateStrategy=MostlyHarmless)

From guardian/actions-riff-raff.

prout-bot commented 7 months ago

Seen on auth, usage, image-loader, metadata-editor, thrall, leases, cropper, collections, media-api, kahuna (merged by @dblatcher 10 minutes and 15 seconds ago) Please check your changes!