search

chanzuckerberg / miniwdl

Workflow Description Language developer tools & local runner
MIT License
177 stars 54 forks source link

[Question]: is runtime container as an array supportted? #683

Open yunhailuo opened 7 months ago

yunhailuo commented 7 months ago

Not sure if I miss the doc somewhere but is miniwdl supporting this spec where

The container key also accepts an array of URI strings. All of the locations must point to images that are equivalent, i.e. they must always produce the same final results when the task is run with the same inputs. It is the responsibility of the execution engine to define the specific image sources it supports, and to determine which image is the "best" one to use at runtime.

?

mlin commented 7 months ago

@yunhailuo "Yes" but it just uses the first one always. Only because we didn't get to making it do something smarter than that. Requirements input is welcome!

yunhailuo commented 7 months ago

Thank you for the quick answer and sorry for my delay.

I have to say I'm not sure how this array should work and didn't found anything specific in SPEC. Our use cases are just same WDL executed under different environment. For example, say we have an AWS env and a local/HPC (which somehow we can use docker) env. So for a task, we are thinking about specifying a private ECR repo and a quay repo.

I do realize this might not be a good example and use case as why we are not just opening up ECR and use that as a single source of truth. But that's our current restriction which we are working on from that direction too. I mean it's likely we won't need this array in the end. It's just I noticed this in SPEC and wondering how that works and when/how to properly use that.