search

lilab-bcb / altocumulus

Command line tool for submitting WDL jobs to Terra or Cromwell server.
https://altocumulus.readthedocs.io
BSD 3-Clause "New" or "Revised" License
7 stars 0 forks source link

Run a local WDL workflow via Terra #57

Open sjfleming opened 1 year ago

sjfleming commented 1 year ago

Is there a way for me to submit a Terra workflow run using a WDL workflow from a local WDL file?

(I would like to be able to test local changes to a WDL without having to upload the WDL to either the Broad methods repo or Dockstore first... because there will probably be errors.)

yihming commented 1 year ago

Currently submitting a local WDL file is only supported in alto cromwell commands, which communicates with a Cromwell server set up by users. To enable the similar feature for Terra, we'll need to check if FireCloud supports it or not.

sjfleming commented 1 year ago

Thanks @yihming, I wonder if I could do what I want using Dockstore.

This question and #56 are both part of the same question, and maybe I can ask you more directly:

I want to use a Terra workspace for continuous integration testing of WDLs in a GitHub repository using GitHub actions. I want a GitHub action to authenticate me to a Terra workspace, where I have test data in the Terra data table, and I want to be able to submit WDL workflows from the GitHub repository, in order to test them.

Is the alto terra run functionality calling the FireCloud API?

yihming commented 1 year ago

Yes, you can find how we call them at https://github.com/lilab-bcb/altocumulus/blob/main/alto/utils/firecloud_utils.py.

sjfleming commented 1 year ago

Thanks @yihming , and while I'm at it, I'll ask one more question: Is there a way to run a WDL defined in a public GitHub repository, instead of a local WDL? Looking at get_dockstore_workflow() https://github.com/lilab-bcb/altocumulus/blob/69a505129f539bf042a32d6bed09cd73db85d4ad/alto/utils/dockstore_utils.py#L42 I noticed that the URL ends up being 

"url": f"https://raw.githubusercontent.com/..."

I can't say I understand the details, but this made me wonder

sjfleming commented 1 year ago

What I mean to say is... a WDL that is on a public GitHub branch, but that has not yet been linked to Dockstore or tagged in Dockstore.

yihming commented 1 year ago

@sjfleming If you are talking about running a WDL by http/https link on Terra, it's not possible yet.

The "url" you saw above is for alto cromwell run command's usage, which submits a job to a user-maintained Cromwell server. You may type alto cromwell run -h and look for the detailed explanation of its -m option. In this command, we allow directly submitting jobs using WDL via public HTTP/HTTPS link.

However, when coming to Terra, i.e. alto terra run command, for WDL published on Dockstore, it is "methodUri" (as below) https://github.com/lilab-bcb/altocumulus/blob/69a505129f539bf042a32d6bed09cd73db85d4ad/alto/utils/dockstore_utils.py#L149 that is used for submission:

https://github.com/lilab-bcb/altocumulus/blob/8151d2a3c653b0e1783ad296536d750d963be0d6/alto/commands/terra/run.py#L120

sjfleming commented 1 year ago

Thanks so much for your answer! That makes sense