Which funnel version?

[jasper1918]

Hi- I'm doing some evaluation tests of cwl platforms and wanted to include cwl-tes. It looks very interesting.

I was able to get the local compute option working but using slurm is giving me a few errors and I believe these are related to my funnel version not being compatible. These could just be configuration items on my end but wanted to make sure I am using compatible versions. I would be grateful for any tips.

Specifically, I see that the argument for funnel has changed --task-id from --taskID.

The outputs are also not found in slurm mode but I do see the outputs exist in WorkDir/{.TaskID}/var/spool/cwl.

Snippet from runtime log:

[job hello.cwl] CREATED TASK MSG----------------------
Task(id=None, state=None, name='hello.cwl', description='', inputs=[], outputs=[Output(url='file:///tmp/tmpxSheKM/foo.bar.ext', path='/var/spool/cwl/foo.bar.ext', type=u'FILE', name='stdout', description=None), Output(url='file:///tmp/tm
pxSheKM/', path='/var/spool/cwl', type=u'DIRECTORY', name='workdir', description=None)], resources=Resources(cpu_cores=None, ram_gb=None, disk_gb=None, preemptible=None, zones=None), executors=[Executor(image='python:2.7', command=['echo
', 'foo.bar'], workdir='/var/spool/cwl', stdin=None, stdout='/var/spool/cwl/foo.bar.ext', stderr=None, env={})], volumes=None, tags={u'CWLDocumentId': 'file:///mounts/isilon/data/eahome/q804348/myprogs/cwl-tes/hello.cwl'}, logs=None, cre
ation_time=None)
[job hello.cwl] SUBMITTED TASK ----------------------
[job hello.cwl] task id: bajgrr4nrqagur2fd8j0
[job hello.cwl] POLLING 'bajgrr4nrqagur2fd8j0'
[job hello.cwl] POLLING 'bajgrr4nrqagur2fd8j0'
[job hello.cwl] POLLING 'bajgrr4nrqagur2fd8j0'
[job hello.cwl] FINAL JOB STATE: COMPLETE ------------------
[job hello.cwl] job error:
Error collecting output for parameter 'output':

cwl-tes --version: cwl-tes 0.2.0 with cwltool 1.0.20180108222053

funnel version: git commit: f211177 git branch: master git upstream: git@github.com:ohsu-comp-bio/funnel.git build date: Tue Jan 16 11:19:33 PST 2018 version: 0.5.0

[adamstruck]

cwl-tes should be able to run against any Funnel version >= 0.4.0.

What does the CWL look like for the above task? What does the resulting funnel task look like?

[jasper1918]

Hi @adamstruck - I was able to jump back in and follow the tasks. In the end I found that my slurm jobs were writing to /tmp on the worker node and hence could could not be found. With that in mind I figured if I specified --tmpdir-prefix to a shared location it would work but it continued to write to /tmp which I suppose is a local worker context. There is another parameter --tmp-outdir-prefix which did modify the output location which I assume has a global context.

Another method that worked was to not specify tmpdir variables and rather use the --cachedir pointed to a shared NAS location. I wanted to put this up so that others can see the behavior of these arguments.

In summary, I was able to get a semi-comprehensive workflow to run local, slurm, and in aws batch configurations--which is awesome. Any updates to docs or usage would also be helpful. Particularly the funnel config files were one point of confusion since variables don't apply on latest versions. Appreciate the help and keep up the great work.

[adamstruck]

Thanks for for the feedback! As far the --tmp-dir-prefix and --tmp-outdir-prefix flag handling, I mostly left that to cwltool codebase to handle. I believe the behaviors you described above are not specific to cwl-tes.

I thought I should direct you to another CWL project I have been contributing to: https://github.com/rabix/bunny

This project offers support for local, google storage, and S3 file systems. I would encourage you to download the latest release and give it a try. Within this release archive check out the core.properties file within the config directory for comments on how to configure it. (you can find it here in the source tree. Some other basic documentation can be found here.

As far as using AWS Batch with Funnel, check out this project: https://github.com/adamstruck/ebsmount

I have been using this project to enable mounting an EBS volume per task. Here are some examples for how that was done: https://github.com/adamstruck/ebsmount/tree/master/resources/funnel

[adamstruck]

Closing due to inactivity.