search

inab / WfExS-backend

Workflow Execution Service Backend
Apache License 2.0
16 stars 6 forks source link

cwl_engine management of arrays of inputs #9

Closed jmfernandez closed 3 years ago

jmfernandez commented 3 years ago

There is an issue with workflow https://raw.githubusercontent.com/kids-first/kf-alignment-workflow/v2.7.3/workflows/kfdrc_alignment_wf.cwl , leading to error message

inputdeclarations.yaml:2:1:  * the `input_bam_list` field is not valid because value is a CommentedMap, expected null or array of <File>

due input_bam_list not being properly represented.

cwl_engine.CWLWorkflowEngine generates file inputdeclarations.yaml before calling cwltool, in order to tell it the input parameters and where to find the files.

That yaml is created by createYAMLFile https://github.com/inab/WfExS-backend/blob/3da5db236f87fbb0d1daea4f084539f3cac8d355/wfexs_backend/cwl_engine.py#L492-L510

which depends on the output from executionInputs https://github.com/inab/WfExS-backend/blob/3da5db236f87fbb0d1daea4f084539f3cac8d355/wfexs_backend/cwl_engine.py#L512-L550

lrodrin commented 3 years ago

added cases to manage arrays of files and strings in executionInputs:

https://github.com/inab/WfExS-backend/blob/92d8dd11f030f4436d609eccb2c13934c450d8da/wfexs_backend/cwl_engine.py#L516-L566

jmfernandez commented 3 years ago

Fixed along several commits. The issue arose when some input or output has more than one possible type. These cases are quite common in CWL, so it needed several refinements. Commits d3363a12df1fafc77ac3441daeb446a103c8ea16 , 2062e81a17dafa78eefc13fae24dbf8095d39b6f , bc0da5f47d094c90775486e90ea9223303bb23fc , 3a75c970a091b1010912ac35b204f04f802b894b , 0f4c029d62bfdf0126a8171efd81861feaf1199a led to the proper handling of this use case