feat(wes): enums for workflow type & type versions

[uniqueg]

Description

Entering free text is generally inconvenient and error prone for users. For the "Workflow type" and "Workflow type version" fields in particular, at any time, only a subset of values is suported (e.g., CWL, SMK for "Workflow Type"). Therefore, it would be easier and safer if users were able to only select among supported workflow types and workflow type versions.

Possible solution(s)

This could easily be implemented via dropdown menus.

However, the more critical part is how the component is to know which workflow types and type versions are supported, as this may be different for each instance, depending on which types and type versions are supported by the WES that the component is connected to. As this information may change from one timepoint to another, the safest way of obtaining this information is through the WES instances GET /service-info endpoint and parsing the (currently) supported workflow types and type versions from the response.

For example, https://prowes.rahtiapp.fi/ga4gh/wes/v1 currently supports the following:

{
  "workflow_type_versions": {
    "CWL": {
      "workflow_type_version": [
        "v1.0"
      ]
    }
  }
}

Alternative context

Therefore, for a WES ELIXIR Cloud Component connected to that WES instance, only the following values should show up in the dropdown menus:

• Workflow type: CWL
• Workflow type version: v1.0

It is also important to note that the available options for the workflow type version are dependent on the workflow type. For example, the service info of a given WES could list the following:

{
  "workflow_type_versions": {
    "CWL": {
      "workflow_type_version": [
        "v1.0"
      ]
    },
    "NFL": {
      "workflow_type_version": [
        "DSL1",
        "DSL2"
      ]
    }
  }
}

Given that the workflow type versions are dependent on the workflow type, the available options for the "Workflow type version" cannot be populated before the "Workflow type" has been selected. Consequently, rather than listing all of v1.0, DSL1 and DSL2, by default no options should be available until the user selects the workflow type first. Then, depending on their selection, the dropdown menu for workflow type versions should be populated (e.g., if the user selects NFL as the workflow type, only then DSL1 and DSL2 should be available in the workflow type version dropdown).

[anuragxxd]

Thanks, Alex for the explanation, approach for this can be as follows:

• Having the select as the field type in the form component design.
• Since the workflow_type_versions is dependent on the value of the workflow_type having the event emit when someone changes workflow_type will solve the issue so that author/user will be able to set the workflow_type_versions. Right now form component only emits event when the form in submitted. Maybe it can also emit the change event whenever the field is changed.
• WES author can then change the workflow_type_versions in fields array accordingly depending on the workflow_type.

[uniqueg]

Thanks @git-anurag-hub. I'm not sure I fully understand the last point. Who is the WES author ad how can they change the workflow_type_versions?

And does your suggested approach pick up changes that may occur on the side of the connected WES, via calling the service info endpoint?

[JaeAeich]

~I can see that we would want dependency between the two fields, but I can't think of a way to abstract that logic into the design/form component. Simplest drops downs would work by passing the array of options to be displayed. Ie~

<Dropdown
    .options=${myOptions}
/>

~But there isn't links/dependecies between other fields with it. We also can't re-render the form based on condition after user has already selected workflow_type because that will erase all the fields that are already entered. If such a case will occur among other APIs as well then I guess it would be worth implementing a custom nested dropdown like the ones we see in when we right~ Wait shoelace has them, something like this should work ig.

[Chaitanya674]

Hello , @JaeAeich and @git-anurag-hub after reading through this issue we need to make a dropdown which shows the options according to the Workflow type selected by making a request to the endpoint it can get the workflow type version ... correct me if I'm wrong we need to make changes in the "lit" , changing it to a input field to a dropdown .. If I'm correct can i try contributing to this ?

[JaeAeich]

yes sure @Chaitanya674!

[Chaitanya674]

Hello ,

I have created the dropdown using the "select" type and like this in the ecc-utils-design .. form.ts file which look like this : I'm facing a little trouble in accessing the form data because we want to first select the type then the version will be listed according to the "Type" we selected for which I need to access the form data.

should i raise a PR to checkout ?

[uniqueg]

Thanks @Chaitanya674. @JaeAeich: Do you wanna answer that?

But, yes, generally it's a good idea to open a PR or draft PR, as it's easier to discuss on code line by line.

[JaeAeich]

@Chaitanya674 BTW, this is deployed proWES. The thing to keep in mind is the config needs to be abstracted. The one way would be to use fields property and have an extra field like selectOptions or something. If we do,

curl -X GET "https://prowes.rahtiapp.fi/ga4gh/wes/v1/service-info" -H "accept: application/json"

We get the

{
...
  "workflow_engine_versions": {
    "additionalProp1": "string",
    "additionalProp2": "string",
    "additionalProp3": "string"
  },
  "workflow_type_versions": {
    "additionalProp1": {
      "workflow_type_version": [
        "string"
      ]
    },
    "additionalProp2": {
      "workflow_type_version": [
        "string"
      ]
    },
    "additionalProp3": {
      "workflow_type_version": [
        "string"
      ]
    }
  }
...
}

Here is the response from another WES implementation,

{
  ...
  "default_workflow_engine_parameters": [
  ...
  ]
  "supported_wes_versions": [
    "1.0.0"
  ],
  "system_state_counts": {
    "CANCELED": 0,
    "CANCELING": 0,
    "COMPLETE": 24,
    "EXECUTOR_ERROR": 1,
    "INITIALIZING": 0,
    "PAUSED": 0,
    "QUEUED": 0,
    "RUNNING": 0,
    "SYSTEM_ERROR": 0
  },
  "workflow_engine_versions": {
    "NFL": "22.10.0",
    "SMK": "6.10.0"
  },
  "workflow_type_versions": {
    "NFL": {
      "workflow_type_version": [
        "22.10.0"
      ]
    },
    "SMK": {
      "workflow_type_version": [
        "6.10.0"
      ]
    }
  }
}

Note how workflow versions, we can fetch this information on ecc-client-ga4gh-wes-runs side and send the data, that being,

const fields: Fields[] = {
...
selectOptions: [
   "NFL": {
      "workflow_type_version": [
        "22.10.0"
      ]
    },
    "SMK": {
      "workflow_type_version": [
        "6.10.0"
      ]
    }
]
...
}

now it should be the job of ecc-utils-design to parse that and render dropdowns with dependencies and what nots. @uniqueg I am pretty sure the workflow_engire_verison and workflow_type_versions are dependent on each other as well, can we run snakemake workflow on an NFL engine? another question would be can there be support for multiple engine version for the same workflow?. Assuming we can't we will have to parse this response before hand, so maybe the files will became something like:

const fields: Fields[] = {
...
selectOptions: [
   "NFL": {
     "available_version: [x, y, z]
      "workflow_type_version": [
        a, b, c
      ]
    },
    ...
]
...
}

But again if you see this pattern would contrain us to 2 levels of nesting, what will happen if in future we have to render 10 levels of interdepedent selects, the above was just a rough idea on the problem, I hope this makes it clear. The main challenge would be to handle multiple interdependency and single responsibility so that rendering and other work is handle by desing not wes package. BTW we have solved such problems of nesting among various package, using different ideas, you can read through the design package, the simplest solution would be an good interface which is fullfills all our requirements to be added to Field interface. Maybe @git-anurag-hub can also weigh in here.

[uniqueg]

Yeah, this is a lot harder than it looks because the underlying data can be really messy...

A few points to consider from real data:

• The workflow type CWL (Common Workflow Language) is a domain-specific language (DSL) for computational analysis workflows that is highly formalized (see here for the specification). It is also (semantically) versioned, i.e., CWL workflows could come in versions like v1.0.2, v1.1.0 or v1.2.0.
• Multiple engine exists that know how to run CWL workflows, each of which typically following their own versioning scheme. A given engine may or may not support a given CWL workflow type version, e.g., engine A may support all CWL v1 major version releases, while engine B may support only v1.0 minor version releases.
• The workflow type SMK (Snakemake) is also a DSL to describe computational analysis workflows. However, it is not strictly formalized and, in itself, is not explicitly versioned.
• Rather, SMK language features evolve continously, together with the Snakemake workflow engine, which is (semantically) versioned. As the snakemake engine is the only workflows engine to run SMK workflows (AFAIK), the Snakemake team decides how the ecosystem progresses. As such, each workflow will be compatible with a range of versions of the Snakemake workflow engine. For example, a workflow could require at least version 6.0.0 of the engine, because it uses features that only snakemake 6.0.0 and newer support. Converselt, it also happens that the snakemake engine drops backwards-compatability with previous versions, and if our workflow uses such a deprecated feature, it will not be compatible with newer version of the engine. So our SMK workflow (which does not really have a workflow_type_version) will require snakemake >= 6.0.0, <8.0.0.
• For the NFL (Nextflow) is a bit of a mix between CWL and Snakemake: Just like with Snakemake, there is a single engine available whose developers determine the future of both the engine and the language together. The Nextflow DSL is also, AFAIK, not formally specified. However, just like CWL, the DSL is nevertheless versioned. Specifically, NFL workflows come in DSL 1 and DSL 2.
• The version ranges of the nextflow engine that support DSL1 and DSL 2 are (I think) well known/defined, making the situation a bit more predictable compared to Snakemake - where compatibility will basically depend on each workflow and the features that are used within it.

Soooo... it's all a bit of a mess!

HOWEVER...

...that's more of a problem for our proWES gateway - which will try to take a workflow and forward it to the most appropriate WES and therefore somehow needs to do that negotiation.

As far as the client is concerned, we currently only need to specify the type and the version - so the combinations aren't endless and can easily be fetched from the /service-info endpoint. Workflow engine versions can be ignored.