Get a working example script for SevenBridges WES

[ianfore]

Started with SBCGC_WES_Example.ipynb.

Posting here as I know others (@mbarkley ) are working on using the SB WES service. I'm modifying the WES client used to submit the GWAS script from the plenary demo that we just merged in.

Currently getting a 415 error. @mattions, do you have any insights on this?

[mattions]

Hi @ianfore we will take a look, in the meantime maybe it makes sense to refer to to the docs also in the Ipython Notebook, so it's easier for people to figure out how to use WES (https://docs.cancergenomicscloud.org/docs/wes-api-overview)

[ghost]

Hi @ianfore

I will look more closely at the code, but here is an example that has worked for me via curl. Note that this is a POST call.

This example uses a workflow found on Dockstore.

set -x

WES_ENDPOINT=https://cgc-ga4gh-api.sbgenomics.com/ga4gh/wes/v1/runs

API_TOKEN="fill me out"

WF_URL=https://dockstore.org/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fkaushik-work%2Felixir-gwas/versions/master/plain-CWL/descriptor/%2Fgwas_workflow.cwl

PROJ="fill me out"

# Change these to files you can access
DRS_METADATA=drs://...
DRS_VARIANTS=drs://...

curl --location --request POST "${WES_ENDPOINT}" \
--header "Content-Type: multipart/form-data" \
--header "X-SBG-Auth-Token: ${API_TOKEN}" \
--form 'workflow_params={
        "project": "'${PROJ}'",
        "use_interruptible_instances": true,
        "inputs": {
            "metadata": {
                "class": "File",
                "path": "'${DRS_METADATA}'"
            },
            "variants": {
                "class": "File",
                "path": "'${DRS_VARIANTS}'"
            }
        }
    };type=application/json' \
--form 'workflow_url='${WF_URL}'' \
--form 'workflow_type=CWL' \
--form 'workflow_type_version=v1.0' \
--form 'tags={}' \
--form 'workflow_engine_parameters={}'

[ianfore]

@mattions Thanks for sharing the link here. Yes that documentation is what I linked to in the notebook and was following through. I started with the example shown there and modified it with the specifics of the files and and workflow that have worked in a non-WES scenario.

@kaushik-work Thanks. Your example has some differences and similarities from my use case. In your first response it looked like you were using DRS URIs in the same way I am - though those URIs don't seem to be there after your edit of the response. As I recall they were BioDataCatalyst URIs. If that's what you used it's reassuring that those URIs worked for you.

You're using a different option for the workflow than I am, so on that aspect of things we may not be comparing apples with apples. Both approaches are supported according to the SB documentation. You're running the workflow from an image in Dockstore. In my case the Docker image is already loaded into the SB environment and I'm referencing it by its id in that environment. Our names for the inputs differ, but that's to be expected as that's defined by the workflows.

I suspect the key difference might be that you're passing the request as a multipart/form rather than as application/json. I'll give that a try.

[ghost]

@ianfore the sbg:// with a path to a workflow will work too. I removed the DRS ids since those files won't be accessible to you. You should replace them with DRS ids of files you can access on the CGC.

Yes. The multipart/form is important.

Thanks.

[ianfore]

Thanks @kaushik-work . Not submitting as a multipart/form was the problem. The notebook is now working. To close it out it would be worth adding retrieval of the results files via DRS. Will take a look at that. It would be a useful illustration of the different authentications that would need to be done to take a workflow from beginning to end. The issue of handling WES outputs has been raised elsewhere.

The need for multipart/form is covered in the WES documentation. It would be useful if the SB WES pages mention this is necessary. Among other things workflow_params has to be passed as application/json as in your example.

[ghost]

@ianfore we will plan to update the documentation, thank you!

[ianfore]

Added download via DRS of the workflow output. See updated notebook. @kaushik-work should we add a separate issue for updating the documentation? If so I'll do so and close this out. Otherwise I'll leave it open and you can close it when done.

[ghost]

Hi @ianfore the documentation has been updated. Thank you.

[ianfore]

Thanks @kaushik-work et al. Closing.