ga4gh / workflow-execution-service-schemas

The WES API is a standard way to run and manage portable workflows.
Apache License 2.0
82 stars 38 forks source link

Separate default engine parameters for different engines in the service info #185

Open vinjana opened 1 year ago

vinjana commented 1 year ago

The API supports multiple workflow engines per WES server. But in the service info the default_workflow_engine_parameters is just a flat list of dictionaries like this

"default_workflow_engine_parameters": [
    {
      "name": "string",
      "type": "string",
      "default_value": "string"
    },
    ...
  ]

Does this mean that all engines must support the same set of parameters with the same defaults? Even if these are very different engines like Nextflow, Snakemake and Toil? This would seem little reasonable.

Wouldn't it make sense to either define separate lists of default parameters for different engines (or even versions)? This could be done in a global default_workflow_engine_parameters structure. E.g.

"default_workflow_engine_parameters": {
   "engineA":  [
        {
          "name": "string",
          "type": "string",
          "default_value": "string"
       },
       ...
    ],
    ...
}

Or it could be done together with some new definition of workflow_engines (following #122):

"workflow_engines": [
    {
      "name": "cwltool",
      "version": "1.0.20181201184214",
      "workflow_types": [
        {
          "language_type": "CWL",
          "language_version": "v1.0"
        }
      ],
      "default_parameters": [
         {
           "name": "string",
            "type": "string",
           "default_value": "string"
        },
        ...
      ]
    }
  ]
patmagee commented 1 year ago

@vinjana another approach that implementations could do is devise their own set of workflow_engine_parameters and map those onto different engines themselves. This does still imply that various engines need to have some sort of property that a given parameter can be mapped to

But since we are starting to have multiple implementations with multiple engines behind the scenes, it is starting to appear that we will need a better strategy for this. I think the second approach you suggest makes a lot of sense. This would allow a way for the API to define the set of parameters that each engine supports independently of each other but also still keep the global default_workflow_engine_parameters and allow the WES api to have it's own set of parameters

uniqueg commented 1 year ago

I also think that the specs should account for this use case and would be very happy to see a PR for this (also preferring option 2 like @patmagee).