search

lanl / BEE

Other
14 stars 3 forks source link

CWL Array Inputs #695

Open jtronge opened 1 year ago

jtronge commented 1 year ago

BEE currently doesn't support CWL array inputs: https://www.commonwl.org/user_guide/topics/inputs.html#array-inputs. This could be useful for scatter-gather-type workflows, or any step that requires a variable number of inputs.

aquan9 commented 5 months ago

After speaking to Steven: We need to add the array to the type map in (https://github.com/lanl/BEE/tree/develop/beeflow/common/parser) parser.py

Some workflows like VASP benefit from having access to this array type. CWL arrays are used for scatter-gather tasks, that will take an array and distribute each element of the array to each task.

Parser generates task objects. In beeflow/common/wf_data.py Task objects are then added to workflow objects. We need to be able to store the arrays as properties of a node in the graph database. Tasks are generated in the graph database using the structure defined in wf_data.py using Cypher code.

Look at the official CWL specification to get details about the requirements for the CWL arrays.