[image-loader] push -> pull based architecture

twrichards commented 7 months ago

Co-authored-by: @dblatcher

Original issue neatly described here https://github.com/guardian/grid/issues/4026, see also the follow-up comments as we explored the problem in the lead up to this PR.

Infrastructure changes in https://github.com/guardian/editorial-tools-platform/pull/733

Change summary by area

/common-lib/**/aws/

new SqsHelpers trait (which helps reading messages from SQS queue, specifically the S3 key)
SimpleSqsMessageConsumer class - more general than the previous SqsMessageConsumer class, which was for an SNS-based queue and is now replaced by the SqsViaSnsMessageConsumer (subclass of the SimpleSqsMessageConsumer)

/common-lib/**/config/

Optional String config for the new infrastructure (ingest SQS queue, ingest S3 bucket, fail S3 bucket).
- sqs.ingest.queue.url
- s3.ingest.bucket
- s3.fail.bucket
The values are optional as the applications must function (using the current process) if the new infrastructure is not in place: IE other Grid users are not required to use queue-based ingestion (yet)

/dev/

new infrastructure is defined and set-up when running in localstack - so the Grid will use queue-based ingestion when running locally
note that localstack does not emulate all of the AWS requirements for Policy statements and permissions, as such the cloudformation template in grid-dev-core.yml does not include the policy configuration needed for the ingest bucket to post messages to the ingest SQS queue

/image-loader/

the ImageLoader Controller continually monitors the SQS queue - on receiving a message, either:
- it ingests the file identified in the message into the Grid, moving the object to the image bucket and deleting the message off the queue,
- the file cannot be processed (eg - invalid mime type) so the object is moved to the ‘fail bucket’ and deleting the message off the queue
- when the instance crashes (eg - out of memory) whilst trying to process a message, the message will go back to the queue (SQS will increment the approximateReceiveCount property of the message every time the message is consumed)
- if the approximateReceiveCount is > 2, the file is transferred directly to the fail bucket - in this case we assume there is a problem with the image file (too big?) that caused the previous instances to crash while trying to process it, so don't try again.
if the queue is configured, healthcheck will fail if the instance has stopped polling the queue for messages
/prepare endpoint : authenticated kahuna users can post the filenames and imageId’s of the images they want to upload and get back the pre-signed URLS to upload the files directly to the s3 ingest queue
new status types for UploadStatusTable:
- ‘Prepared’ = the UI has requested and been sent a presigned URL
- ‘Queued’ = the UI has reported that the upload to the queue bucket has finished (a condition in the ‘updateStatus’ method so can only transition to ‘Queued’ if status was ‘Prepared’

/kahuna/

Config: Kahuna will continue to use the old/current method of posting image files directly to ImageLoader if there is no queue-based ingest bucket defined , otherwise, it will:

generate the mediaId (SHA-1 hash of the file) for each image client side (there is a new dependency, filehash for this)
POST the filenames and media ids of the files to be uploaded to the /prepare endpoint to receive the presigned URL for each image
PUT the file in S3 using the the presigned URL
On receiving an ‘OK’ response from S3, POST to /uploadStatus endpoint to update the image status ‘Queued’
(As before) poll the /uploadStatus endpoint to GET the image status, updating the UI when an image is ‘Completed’ or ‘Failed’

Diagrams

Architecture

reveal mermaid code

```mermaid flowchart TD user(Kahuna) ftp(FTP server) feeds(Grid Feeds) image-loader(image-loader\nec2) ingest[(ingest bucket\ns3)] queue(SQS queue) bucket[(image bucket\ns3)] elastic[(elasticsearch)] status[(upload status\ndynamo)] user <-- request\n presigned URL --> image-loader user -- upload file\n(presigned URL) -->ingest ftp -- upload file--> ingest feeds -- upload file--> ingest ingest -- objectCreated --> queue queue <-- pulls messages --> image-loader image-loader -- move file from\n ingest bucket---> bucket image-loader -- add image ---> elastic image-loader -- update status ---> status ```

UI upload sequence

sequenceDiagram
    participant MediaAPI
    Kahuna->>Kahuna: Generate mediaId (SHA-1)
    Kahuna->>Image Loader: POST prepare (Map of id -> filename)
    Image Loader->>Image Loader: upload status: Prepared
    Image Loader-->> AWS: GeneratePresignedUrlRequest
    AWS-->> Image Loader: GeneratePresignedUrlResponse
    Image Loader->>Kahuna: Map of id -> presigned url
    Kahuna-->>AWS: PUT file in bucket 
    AWS-->>Kahuna: upload confirmed
    Kahuna->>Image Loader: upload status: Queued
    Kahuna-->Image Loader: Poll: uploadStatus
    Kahuna->>Kahuna: Update UI
    AWS-->>Image Loader: SQS message - s3:ObjectCreated
    Image Loader->>Image Loader: Ingest image, upload status: Completed
    Kahuna-->Image Loader: Last Poll: uploadStatus
    Kahuna->>Kahuna: Update UI
    Kahuna-->MediaAPI: Poll: image details
    Kahuna->>Kahuna: Update UI

github-actions[bot] commented 7 months ago

Deploy build 12208 to TEST

All deployment options

- [Deploy build 12208 to TEST](https://riffraff.gutools.co.uk/deployment/deployAgain?project=media-service%3A%3Agrid%3A%3Aall&build=12208&stage=TEST&updateStrategy=MostlyHarmless&action=deploy) - [Deploy parts of build 12208 to TEST by previewing it first](https://riffraff.gutools.co.uk/preview/yaml?project=media-service%3A%3Agrid%3A%3Aall&build=12208&stage=TEST&updateStrategy=MostlyHarmless)

From guardian/actions-riff-raff.

prout-bot commented 6 months ago

Seen on image-loader (merged by @twrichards 8 minutes and 51 seconds ago) Please check your changes!

prout-bot commented 6 months ago

Seen on auth, usage, metadata-editor, thrall, leases, cropper, collections, media-api, kahuna (merged by @twrichards 9 minutes and 56 seconds ago) Please check your changes!

guardian / grid