search

dockstore / dockstore

An app store for scientific workflows, tools, notebooks, and services
https://dockstore.org/
Apache License 2.0
120 stars 28 forks source link

cwltool is not quite working with direct launch on workflows #2300

Closed denis-yuen closed 5 years ago

denis-yuen commented 5 years ago

Bug Report

Expected behaviour

Launch instructions with cwltool should just work

https://dockstore.org/workflows/github.com/dockstore-testing/md5sum-checker:develop?tab=launch

Actual behaviour

With workflows using multiple files (as opposed to tools or workflows in one file)

$ cwl-runner https://dockstore.org:443/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fdockstore-testing%2Fmd5sum-checker/versions/develop/plain-CWL/descriptor/%2Fmd5sum%2Fmd5sum-workflow.cwl Dockstore.json

works

$ cwltool github.com/dockstore-testing/md5sum-checker:develop Dockstore.json
$ cwltool --make-template github.com/dockstore-testing/md5sum-checker:develop > input.yaml

These two do not work. looks like they don't resolve the #workflow and md5sum/md5sum-workflow.cwl fragments. We can do part of this by generating:

cwltool \#workflow/github.com/dockstore-testing/md5sum-checker:develop Dockstore.json

But the resolver still needs the initial file path

┆Issue is synchronized with this Jira Bug ┆Fix Versions: Dockstore 1.7 ┆Issue Number: DOCK-627 ┆Sprint: Backlog ┆Issue Type: Bug

denis-yuen commented 5 years ago

Will need to add cwltool version for direct launch after PR to cwltool https://github.com/common-workflow-language/cwltool/pull/1093

coverbeck commented 5 years ago

I went to https://staging.dockstore.org/workflows/github.com/dockstore-testing/md5sum-checker:develop?tab=launch and successfully ran the first 3 launch examples (dockstore convert, dockstore launch, and cwl-runner) on Ubuntu. Skipped the wget one.

The cwl-runner one fails on a Mac, but I think that's just a Mac/Docker issue.

The checker commands seem out of order -- the second one describes how to complete a template JSON, the first one describes how to run it. Since you need JSON to run it, seems like it should be the other way around.

I was able to get both checker commands to run.

@denis-yuen: Is this sufficient verification? Not entirely clear from the description.

Also is the order of the checker commands wrong, or am I misunderstanding? I will log a seprarate ticket if so.

Not sure if I'm misunderstanding stuff, so putting comment here for further discussion.

denis-yuen commented 5 years ago

@coverbeck this ticket is for running directly with cwltool without using the dockstore CLI so cwl-runner https://staging.dockstore.org/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fdockstore-testing%2Fmd5sum-checker/versions/develop/plain-CWL/descriptor/%2Fmd5sum%2Fmd5sum-workflow.cwl Dockstore.json would count (but seemed to be broken for you on a Mac)

https://github.com/common-workflow-language/cwltool#use-with-ga4gh-tool-registry-api has more info (same syntax).

Depending on when it broke on your Mac, it should be ok if cwltool launched, downloaded descriptors, and started Docker properly. Alternatively, try on a VM?

coverbeck commented 5 years ago

cwl-runner fails on a Mac because The paths /private/tmp/docker_tmpb1v2fkvt and /private/tmp/docker_tmplvfnp9i1 are not shared from OS X and are not known to Docker, which is a Mac limitation. The command does work on Ubuntu.

FYI, when running with Dockstore CLI on a Mac it works because we explicitly set the --tmpdir-prefix when invoking cwl-runnner. Or that's my guess.

Will log the order of launch commands as a separate ticket.

Verified with 1.7.0-rc.0 on staging.