hello world in local mode fails to execute

[rivershah]

I installed dsub and attempting the hello world example. here are the commands I've tried:

pip install sub
dsub --version
dsub version: 0.4.2
 dsub \
   --provider local \
   --logging /tmp/dsub-test/logging/ \
   --output OUT=/tmp/dsub-test/output/out.txt \
   --command 'echo "Hello World" > "${OUT}"' \
   --wait

JobExecutionError: One or more jobs finished with status FAILURE or CANCELED during wait.

2020-11-07 12:29:22 I: Localizing inputs.
2020-11-07 12:29:22 I: Checking image userid.
2020-11-07 12:29:22 I: Task image: ubuntu:14.04
2020-11-07 12:29:23 I: Running Docker image.
2020-11-07 12:29:23 I: Docker exit code 0.
2020-11-07 12:29:23 I: Ensure host user (501:20) owns Docker-written data
2020-11-07 12:29:24 I: Copying outputs.
2020-11-07 12:29:24 E: Error on or near line 45; /var/folders/0v/x5pftd6d6_v9ylh2rnjsn8f80000gn/T/dsub-local/hello-worl--xxxxx--201107-122922-75/task/runner.sh exiting with status 127
2020-11-07 12:29:24 I: Copying the logs before cleanup

Any assistance will be much appreciated.

[mbookman]

Hi @RiverShah !

Thanks for reporting this. This seems to be a problem specific to MacOS. The use of /var/folders for temporary files instead of /tmp looks to be a new (ish?) development.

I think the problem is simply that at the very end, the local provider is trying to copy outputs to /tmp/dsub-test/output/, but that this directory does not exist. The "hello world" example assumes that TMPDIR is always /tmp. So you can either solve this by setting TMPDIR to /tmp or probably more appropriate would be for the "hello world" example to be something like:

dsub \
   --provider local \
   --logging "${TMPDIR:-/tmp}/dsub-test/logging/" \
   --output OUT="${TMPDIR:-/tmp}/dsub-test/output/out.txt" \
   --command 'echo "Hello World" > "${OUT}"' \
   --wait

which indicates that the output and logging should go to the TMPDIR or if not set, then go to /tmp.

[rivershah]

@mbookman Your suggested changes make the example work. Thank you for this. I suspect this is due to changes in OS Catalina (10.15.7) Ironically, I had already gotten toy examples for the cloud running and managed to run some larger workflows with the google-cls provider. On another note, from a long time user of SGE, dsub with the local and google compute engine providers is incredibly easy pathway to scale out compute. Thanks to all the contributors for writing and maintaining this great tool.