DataBiosphere / dsub

Open-source command-line tool to run batch computing tasks and workflows on backend services such as Google Cloud.
Apache License 2.0
265 stars 44 forks source link

Allow use of command and script flags at the same time? #187

Open slagelwa opened 4 years ago

slagelwa commented 4 years ago

I have a pretty complicated command that I want to run on a number of files, each on their own vm. So I put the command in a bash script and I put my file names in a tsv file. What I wanted to run was:

dsub --tasks myfiles --script ./messycmd --command './messycmd $FILE'

However dsub doesn't currently allow you to provide both a script and a command. I can always whip up a docker with my script in it and run it that way, but it just seems kinda unnecessary.

wnojopra commented 4 years ago

Hi @slagelwa,

I might be misunderstanding something, but I'm not sure why you'd need both --command and --script options. You should be able to call your complicated command in its entirety in your script. Within your script, you are able to reference the $FILE for the task. You may find this doc on scripts and commands helpful. There is also an example available here that runs a complicated script on multiple files, each on its own vm.

slagelwa commented 4 years ago

Your correct I can, but I'd rather my script read its input from a command line argument vs. having an embedded environment variable. That way if I want I can run it from the command line instead of as a dsub job. I understand I can set the environment variable and run it, but that's a little more unorthodox.