Consider implementing the Common Workflow Language standard

[mr-c]

https://www.commonwl.org

[jackywu]

+1

[xianlubird]

+1

[zanieb]

+1

Would be willing to contribute to this.

[cuericlee]

maybe we could consider to parse both wdl/cwl into DAG and parse inputs as parameters in argo, take NAS volume as an example PoC, then go object storage.

https://github.com/gatk-workflows/broad-prod-wgs-germline-snps-indels/blob/master/JointGenotypingWf.wdl https://github.com/gatk-workflows/broad-prod-wgs-germline-snps-indels/blob/master/JointGenotypingWf.hg38.inputs.json

[bbrauzzi]

+1

[alexec]

This is unlikely to be developed by the core team at this time - it is a perfect piece of work for community contribution.

[openbioinfo]

+1

[alexec]

I wonder if a transpiler from CWS to Argo YAML would solve this.

[isubasinghe]

Picked this up, going to be developing on a CWL to argo transpiler on my fork https://github.com/isubasinghe/argo-workflows

[isubasinghe]

Hey @mr-c would you by any chance have time for a quick chat about CWL? Perhaps we could setup a small zoom meeting?

[isubasinghe]

Thanks I have booked a meeting for Monday morning, looking forward to chatting with you

[alexec]

See #7503, #7506, #7538, #7553 ,#7578

[isubasinghe]

Apologies for not getting back on to this, my latest work is available on my branch here: https://github.com/isubasinghe/argo-workflows/tree/feat/cwl2argo-output/cmd/cwl2argo

It is capable of compiling "CommandLineTool" types but it would be trivial to reuse the functions to make the compiler work for "Workflow" types.

Unfortunately I am busy with university and settling in another country at the moment so I haven't had time to work on Argo Workflows.

[alexec]

Would anyone watching this issue like to pick this up?

[JavascriptMick]

+1

[matbun]

+1

[gerard0315]

+1

[SerRichard]

@alexec Hi, I am happy to pick this up, it's come up at work a couple of times over the last year.

Would you consider a PR where this is encapsulated as a new argo cli command? Something like, argo transpile FILE ?

[isubasinghe]

Hey @SerRichard, I'd be happy to help you out with this. I have a parser for CWL that would help you. It already supports some amount of argo.

It's unfortunately not as simple as I personally thought it was at the time.

[SerRichard]

@isubasinghe The code in the fork, or something else? & do you remember the sticking points you found when trying to work on it before?

[mr-c]

For parsing CWL in Go, I recommend adding Go support to our code generator. It is already used to make CWL parsers for C++, Python, Java, Typescript, .Net, and soon also Rust:

https://schema-salad.readthedocs.io/en/latest/#codegen-examples

(I can provide guidance and mentoring for this task)

[isubasinghe]

@SerRichard Yeah the fork has an additional cli, I think one of the branches are pretty much complete for a CWL type as well.

Probably not much for you to do from a parser and type checking point of view. I can't remember the exact pain points, but generally speaking the correspondence between CWL and argo isn't one to one and requires some creativity to address.

My tool already does some generation, this: https://github.com/isubasinghe/argo-workflows/blob/feat/cwl2argo-output/cmd/cwl2argo/argo_simple.yaml was generated from https://github.com/isubasinghe/argo-workflows/blob/feat/cwl2argo-output/cmd/cwl2argo/locations.json and https://github.com/isubasinghe/argo-workflows/blob/feat/cwl2argo-output/cmd/cwl2argo/hello.cwl

[SerRichard]

@isubasinghe @mr-c Sorry for the delay replying, I've only now had a chance to look at this again. Going through the cwl spec there's a couple of places where I'm not certain what the expected behaviour of a cwl2argo parser would be.

Firstly, take the example of a CommandLineTool where no docker requirement is defined, or only software requirements are defined. Install commands could be added to a step's script in a workflow, but in both cases I would still expect the user to define some base environment that this cli is expected to run in. So could Docker requirements with a docker pull be a requirement for the parser? If not, do you have any other ideas on how to handle this?

Secondly, I don't totally grasp how to map cwl mounts to either argo workflow mounts or artifacts. For example, when a Dirent in cwl is specified, the files must be staged before execution. If that is the case, then the argo workflow should reference either a mount or artifact. Is there a way to infer from cwl which syntax ought to be used when parsing to an argo worflow? If not, then I'm not sure how to get away from asking a user to parse a flag when calling the cwl2argo parser.

Any input is appreciated!

[mr-c]

So could Docker requirements with a docker pull be a requirement for the parser?

Yes, that is reasonable, and true for basically all the cloud-based CWL compatible platforms 👍

Secondly, I don't totally grasp how to map cwl mounts to either argo workflow mounts or artifacts. For example, when a Dirent in cwl is specified, the files must be staged before execution. If that is the case, then the argo workflow should reference either a mount or artifact. Is there a way to infer from cwl which syntax ought to be used when parsing to an argo worflow? If not, then I'm not sure how to get away from asking a user to parse a flag when calling the cwl2argo parser.

I'm not clear on the question here. Can you share some examples?