search

SPARC-FAIR-Codeathon / sparc-flow

A Python tool to describe and run tools and workflows for processing SPARC datasets in accordance with FAIR principles.
Apache License 2.0
4 stars 1 forks source link

API - Design #2

Closed chinchien-lin closed 1 year ago

chinchien-lin commented 1 year ago

Design and create generic API to help create tools and workflows easily e.g. in CWL format (see if something exists first and reuse)

frenchmatthew commented 1 year ago

Prasad and Chinchien initial discussions.

Aim: Different languages for creating tools, but we want to agnostic to the language.

Sudo-code usage (CWL language example): 

-------- Creating and running --------- 
# create a workflow
workflow = sparc_flow.workflow(language=cwl)      

# add step
step = sparc_flow.step()
step.define_input("hello")  
step.define_tool()   
step.define_output()  

workflow.add_step()   
workflow.save()  # save cwl file
workflow.run()    
workflow.generate_sds() # bundles output data with meta-data (e.g. image array with header file - affine transform, spacing, etc)   
                                        # Look at SPARC data-structure 

------------------------------- 
---------- Creating a tool ----------

tool = sparc_flow.Tool()  
toool.define_input()  
tool.define_execcutable()  
tool.define_output()  
too.generate_sds()

--------------------------------

Next Steps:

Look for python tools to programmatically create tools and workflow descriptions for CWL language.
Create basic API entries.
Look at SPARC data-strcuture.
Run tutorial for creating a SPARC dataset structure - Tutorial 2 in README of sparc-me

Look into:

  1. https://cwl-utils.readthedocs.io/en/latest/
  2. https://pypi.org/project/cwltool/
frenchmatthew commented 1 year ago

Next step 1 completed:
We will use scriptcwl to create tools and workflow descriptions for CWL language.
Full documentation found here.

chinchien-lin commented 1 year ago

Hi @frenchmatthew

below is the Updated API design

Tool tool = sparc_flow.Tool() tool.set_command() tool.set_arguments([array of strings]) e.g. ["-m", "path/to/file/or/script/without/extension"] tool.set_input_type("int") tool.set_output_type("file") tool.set_output_path("output.txt") tool.generate_description() tool.generate_sds()

Workflow workflow = sparc_flow.Workflow() workflow.set_steps(tool_paths=["tool 1 path"])

workflow.set_input_value(262) workflow.generate_description() workflow.generate_parameters(file_type=json default argument) workflow.generate_sds()

workflow.run()