search

umccr / orcabus

 🐋 UMCCR Pipeline & Workflow Orchestration
4 stars 0 forks source link

Deprecate icav2-copy-batch-utility #293

Closed alexiswl closed 5 months ago

alexiswl commented 6 months ago

ICAv2 Copy Batch Utility as a stack is overkill.

As @reisingerf suggested, each service that requires the copy batch utility can generate their own.

Create a copy batch utility as a construct and then have microservices create their own versions of this construct.

victorskl commented 6 months ago

(made me shock momentarily first)

Create a copy batch utility as a construct

Composing it as a construct sound good.! 👍

alexiswl commented 6 months ago

It does mean that the cttsov2 service will have five step functions associated with its stack

victorskl commented 6 months ago

Hear you. Hang in there. Pls yell, if step functions started to become out of hand. We can always regroup and discuss on tech choices...

brainstorm commented 6 months ago

This decision makes sense, in whatever meeting that happened, I've been thinking the same while working on the ICAv1 copy batch utility stack, more generic constructs are needed and I reckon that AWS S3 Batch Ops can do a good job at that.

alexiswl commented 6 months ago

in whatever meeting that happened

No meeting, just orcabus daydreams in my head.

One other benefits is that the step functions executions are less cluttered with different services. If I want to see when the cttso executed the copy batch I can easily find executions of this without needing to trawl through the bssh fastq copy executions (because they're different step functions now with different names).