Slurm: better catch failures

[tiborsimko]

When testing Slurm workflow execution, I noticed the jobs fail with:

$ sacct
468744.0     singulari+                                1     FAILED    127:0
$ tail -3 ./reana_job.468744.err
FATAL:   container creation failed: mount /hpcscratch/user/simko->/var/singularity/mnt/session/hpcscratch/user/simko error: while mounting /hpcscratch/user/simko: while getting mount flags for /hpcscratch/user/simko: while searching parent mount point entry for /hpcscratch/user/simko: no parent mount point found
srun: error: hpc003: task 0: Exited with exit code 255
srun: Terminating job step 468744.0

This made the workflow pod on Kubernetes to stop, but the workflow status seems still running:

$ reana-client status -w slurm
NAME    RUN_NUMBER   CREATED               STATUS    PROGRESS
slurm   2            2020-01-07T16:53:00   running   0/3   

We should:

• better catch errors and update workflow status appropriately
• se why the singularity command fails now; it was working well around December

[roksys]

It's not slurm related. I just did a test on Kubernetes with the first job of roofit example exiting with 1 and got the same status.

$ reana-client status -w workflow.7
NAME       RUN_NUMBER   CREATED               STATUS    PROGRESS
workflow   7            2020-01-09T14:48:54   running   0/3