Closed mturilli closed 8 years ago
Fail is a relative term here :-)
Whats the return code of the task supposed to be?
I took the liberty to look into your sandbox (...016821.0012-pilot.0000) and see:
2016-01-21 06:22:02,261: agent_0.AgentExecutingWatcher_POPEN.0: agent_0.AgentExecutingComponent_POPEN.0: Watcher : INFO : Unit unit.000004 has return code 32.
Which RP will translate rightfully to fail.
Here the radical_pilot_cu_launch_script.sh
for a CU:
#!/bin/bash -l
echo script start_script `/work/02855/mturilli/radical.pilot.sandbox/rp.session.Matteos-MacBook-Pro.local.mturilli.016821.0024-pilot.0000/gtod` >> /work/02855/mturilli/radical.pilot.sandbox/rp.session.Matteos-MacBook-Pro.local.mturilli.016821.0024-pilot.0000/unit.000000/PROF
# Change to working directory for unit
cd /work/02855/mturilli/radical.pilot.sandbox/rp.session.Matteos-MacBook-Pro.local.mturilli.016821.0024-pilot.0000/unit.000000
echo script after_cd `/work/02855/mturilli/radical.pilot.sandbox/rp.session.Matteos-MacBook-Pro.local.mturilli.016821.0024-pilot.0000/gtod` >> /work/02855/mturilli/radical.pilot.sandbox/rp.session.Matteos-MacBook-Pro.local.mturilli.016821.0024-pilot.0000/unit.000000/PROF
# Environment variables
export RP_SESSION_ID=rp.session.Matteos-MacBook-Pro.local.mturilli.016821.0024 RP_PILOT_ID=pilot.0000 RP_AGENT_ID=agent_0 RP_SPAWNER_ID=agent_0.AgentExecutingComponent_POPEN.0.child RP_UNIT_ID=unit.000000
# The command to run
task "serial" "1" "30.0" "65536" "65536" "0" "0" "0"
RETVAL=$?
echo script after_exec `/work/02855/mturilli/radical.pilot.sandbox/rp.session.Matteos-MacBook-Pro.local.mturilli.016821.0024-pilot.0000/gtod` >> /work/02855/mturilli/radical.pilot.sandbox/rp.session.Matteos-MacBook-Pro.local.mturilli.016821.0024-pilot.0000/unit.000000/PROF
# Exit the script with the return code from the command
exit $RETVAL
Here the command executed on my login shell:
$ task "serial" "1" "30.0" "65536" "65536" "0" "0" "0"
serial: 9
sleep interval: 30.000000
$
Both with a pipe or checking $?
after task
has exited returns 0
A quick internet search of "exit code 32" seems to point to FS issues. That would correlate to the empty log files too ...
Darn (Homer TM)! Is there a way to tell RP to use my home directory for testing whether the issue is with $WORK
? Suggestions for different approaches more than welcome. I wonder though why the very same run with some input/output files presents no issues...
I can't explain that difference to be honest, so in that sense it was just a long shot, not an explanation.
Look at this one for more puzzlement? ;)
login2.stampede(24)$ sh
login2.stampede(3)$ task "serial" "1" "30.0" "65536" "65536" "0" "0" "0"
serial: 9
sleep interval: 30.000000
login2.stampede(4)$ echo $?
32
Ok, so the "task" is broken?
Well, before it was returning 0
. Since when I started a new shell, it returns 32
. Puzzling.
The issue was solved by patching task.c from the Skeleton repository forcing it to exit with a 0 code when finishing to execute successfully.
RP 0.38 installed from pip
Running a Bo 8 T, 1 core, 1 minute on Stampede. The kernel of each task is the skeleton executable. I successfully tested the execution line of each CU on Stampede. The CUs do not fail and indeed STDERR of each CU is empty.
When specifying staging directives CUs successfully execute:
Example of a CU description:
Pilot description:
The following CUs instead fail:
With (almost) the same pilot description:
Note the DEBUG output where I print also STDOUT and STDERR for each CU, confirming that no error is reported:
Andre gained access to my sandbox on Stampede but noticed nothing particularly wrong with the execution. Happy to provide pointer to the sanbox location if useful.