search

wellcomecollection / catalogue-pipeline

:oil_drum: The data pipeline services extracting & transforming data from our museum and collections.
https://developers.wellcomecollection.org/catalogue
MIT License
13 stars 2 forks source link

Re-architect the batcher service so that it works as a Lambda #2721

Open kenoir opened 1 week ago

kenoir commented 1 week ago

Part of https://github.com/wellcomecollection/platform/issues/5784

This is the specific task of refactoring the ECS service so that it can operate as an SQS triggered Lambda.

It should be preceded by the RFC described in the linked issue, and followed by an ADR describing the change made.

paul-butcher commented 6 days ago

One of the things I want us to consider is that the whole SQS-in/SNS-out business has its tentacles all through the invocation part of this application. Ideally, we should be able to invoke it not just by Lambda (which would still be SQS), but also from a CLI by passing it a list of strings (probably on STDIN), without having to run pretend AWS services.

Throughout the whole pipeline, we have big chains of Akka stream stuff that might be better, both for readability and reuse, if we crack them apart and name them, preferably restricting the interface parts (SQS/SNS or STDIO or whatever) to the very periphery.