vivek-bala
commented
10 years ago Could you run them again with RADICAL_PILOT_VERBOSE set to DEBUG please and paste the output ?
export RADICAL_PILOT_VERBOSE=DEBUGClosed CharlieLaughton closed 10 years ago
vivek-bala
commented
10 years ago Could you run them again with RADICAL_PILOT_VERBOSE set to DEBUG please and paste the output ?
export RADICAL_PILOT_VERBOSE=DEBUG
CharlieLaughton
commented
10 years ago Hi Vivek,
Here is the output from the Amber/CoCo run. It’s rather long, so I will send the output from the Gromacs/LSDMap run in a separate message.
Charlie
2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] radical.pilot version: 0.19 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] using database url mongodb://localhost:27017/ 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] using database name radicalpilot 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/das4.json 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/epsrc.json 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/futuregrid.json 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/iu.json 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/localhost.json 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/lrz.json 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/ncar.json 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/xsede.json 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] New Session created{'database_url': 'mongodb://localhost:27017/', 'database_name': 'radicalpilot', 'last_reconnect': None, 'uid': '54256ce9d3259d2b071d1655', 'created': datetime.datetime(2014, 9, 26, 13, 40, 57, 780571)}. 2014:09:26 14:40:57 radical.pilot.MainProcess: [DEBUG ] Worker thread (ID: Thread-1[4464848896]) for PilotManager 54256ce9d3259d2b071d1656 started. 2014:09:26 14:40:57 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for PilotManager 54256ce9d3259d2b071d1656. 2014:09:26 14:40:57 radical.pilot.MainProcess: [DEBUG ] saga.utils.PTYShell ('sftp://stampede.tacc.utexas.edu/') 2014:09:26 14:40:57 radical.pilot.MainProcess: [DEBUG ] PTYShell init <saga.utils.pty_shell.PTYShell object at 0x1099eead0> 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] PTY prompt pattern: [\$#%>]]\s*$ 2014:09:26 14:40:57 radical.pilot.MainProcess: [DEBUG ] open master pty for [ssh] [laughton@stampede.tacc.utexas.edu] laughton: /usr/bin/env TERM=vt100 "/usr/bin/ssh" -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=yes -o ControlPath=/tmp/saga_sshcharlie%h_%p.laughton.ctrl laughton@stampede.tacc.utexas.edu' 2014:09:26 14:40:57 radical.pilot.MainProcess: [DEBUG ] PTYProcess init <saga.utils.pty_process.PTYProcess object at 0x1099eec50> 2014:09:26 14:40:57 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=yes -o ControlPath=/tmp/saga_sshcharlie%h%p.laughton.ctrl laughton@stampede.tacc.utexas.edu 2014:09:26 14:40:57 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 33](export PS1='$' ; set prompt='$'n) 2014:09:26 14:40:58 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 51](export PS1='$' > /dev/null 2>&1 || set prompt='$'n) 2014:09:26 14:40:58 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 28](printf 'HELLO%dSAGAn' 1n) 2014:09:26 14:40:59 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 51](export PS1='$' > /dev/null 2>&1 || set prompt='$'n) 2014:09:26 14:40:59 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 28](printf 'HELLO%d_SAGAn' 2n) 2014:09:26 14:41:00 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 1023](Last login: Fri Sep 26 08:39:1 ... guides/nUser News: http:) 2014:09:26 14:41:00 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 1024](//www.tacc.utexas.edu/user-ser ... has three parallel file system)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>) 2014:09:26 14:41:00 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 350](s: $HOME %28permanent,n quota'd, backed-up%29 $WORK %28permanent, quota'd, not backed-up%29 andn $SCRATCH %28high-speed purged storage%29. The "cdw" and "cds" aliasesn are provided as a convenience to change to your $WORK and $SCRATCHn directories, respectively.n__nn) 2014:09:26 14:41:00 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 243](--------------------- Project balances for user laughton ----------------------n| Name Avail SUs Expires | |n| TG-MCB090174 676178 2014-09-30 | |n) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 405](------------------------ Disk quotas for user laughton ------------------------n| Disk Usage %28GB%29 Limit %Used File Usage Limit %Used |n| /home1 1.1 5.0 21.18 21903 150000 14.60 |n| /work 223.8 1024.0 21.85 7963 3000000 0.27 |n-------------------------------------------------------------------------------n) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 173](nTip 110 %28See "module help tacc_tips" for features or how to disable)\n\n Bash user can find out all the locations of the foo command by doing:\n $ type -a foo\n\n) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 53](login1.stampede%281%29$ $$HELLO_1_SAGAn$$HELLO_2_SAGAn$) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] got shell prompt trigger (4) ( See "man slurm" or the Stampede user guide for more detailed information.
--> To see all the software that is available across all compilers and mpi stacks, issue: "module spider"
--> To see which software packages are available with your currently loaded compiler and mpi stack, issue: "module avail"
--> Stampede has three parallel file systems: $HOME (permanent, quota'd, backed-up) $WORK (permanent, quota'd, not backed-up) and $SCRATCH (high-speed purged storage). The "cdw" and "cds" aliases are provided as a convenience to change to your $WORK and $SCRATCH directories, respectively.
--------------------- Project balances for user laughton ---------------------- | Name Avail SUs Expires | | | TG-MCB090174 676178 2014-09-30 | | ------------------------ Disk quotas for user laughton ------------------------ | Disk Usage (GB) Limit %Used File Usage Limit %Used | | /home1 1.1 5.0 21.18 21903 150000 14.60 |
Tip 110 (See "module help tacc_tips" for features or how to disable)
Bash user can find out all the locations of the foo command by doing: $ type -a foo
login1.stampede(1)$ $$HELLO_1_SAGA $$HELLO_2_SAGA) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] got initial shell prompt (5) ( $) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] Got initial shell prompt (5) ( $) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] PTYProcess init <saga.utils.pty_process.PTYProcess object at 0x1099eecd0> 2014:09:26 14:41:01 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshcharlie%h_%p.laughton.ctrl laughton@stampede.tacc.utexas.edu 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] write: [ 8] [ 33](export PS1='$' ; set prompt='$'n) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 1023](Last login: Fri Sep 26 08:41:0 ... guides/nUser News: http:) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 1024](//www.tacc.utexas.edu/user-ser ... has three parallel file system)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] running command shell: exec /bin/sh -i 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] write: [ 8] [ 47](stty -echo ; unset HISTFILE ; exec /bin/sh -in) 2014:09:26 14:41:01 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 350](s: $HOME %28permanent,n quota'd, backed-up%29 $WORK %28permanent, quota'd, not backed-up%29 andn $SCRATCH %28high-speed purged storage%29. The "cdw" and "cds" aliasesn are provided as a convenience to change to your $WORK and $SCRATCHn directories, respectively.n__nn) 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 243](--------------------- Project balances for user laughton ----------------------n| Name Avail SUs Expires | |n| TG-MCB090174 676178 2014-09-30 | |n) 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 405](------------------------ Disk quotas for user laughton ------------------------n| Disk Usage %28GB%29 Limit %Used File Usage Limit %Used |n| /home1 1.1 5.0 21.18 21903 150000 14.60 |n| /work 223.8 1024.0 21.85 7963 3000000 0.27 |n-------------------------------------------------------------------------------n) 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 244](nTip 10 %28See "module help tacc_tips" for features or how to disable)\n\n Bash does not automatically source your .bashrc on login; if you want this behavior, source your .bashrc from .profile or the login startup script of your choice.\n\n) 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 21](login1.stampede%281%29$ $) 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] write: [ 8] [ 100](unset PROMPT_COMMAND ; unset HISTFILE ; PS1='PROMPT-$?->'; PS2=''; export PS1 PS2 2>&1 >/dev/nulln) 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] read : [ 8] 1 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] read : [ 8] 10 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] got new shell prompt 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] run_sync: echo "WORKDIR: $PWD" 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] write: [ 8] [ 21](echo "WORKDIR: $PWD"n) 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 42](WORKDIR: /home1/01915/laughtonnPROMPT-0-) 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Determined remote working directory for sftp://stampede.tacc.utexas.edu/: '/home1/01915/laughton' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] PTYShell del <saga.utils.pty_shell.PTYShell object at 0x1099eead0> 2014:09:26 14:41:02 radical.pilot.MainProcess: [INFO ] Launching ComputePilot {u'state': u'PendingLaunch', u'commands': [], u'description': {u'project': u'TG-MCB090174', u'resource': u'stampede.tacc.utexas.edu', u'queue': u'normal', u'sandbox': None, u'cleanup': None, u'pilot_agent_priv': None, u'memory': None, u'cores': 64, u'runtime': 60}, u'sagajobid': None, u'started': None, u'cores_per_node': None, u'output_transfer_started': None, u'finished': None, u'submitted': datetime.datetime(2014, 9, 26, 13, 41, 2, 744000), u'output_transfer_finished': None, u'sandbox': u'sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256ce9d3259d2b071d1657/', u'pilotmanager': u'54256ce9d3259d2b071d1656', u'unitmanager': None, u'heartbeat': None, u'statehistory': [{u'timestamp': datetime.datetime(2014, 9, 26, 13, 41, 2, 742000), u'state': u'PendingLaunch'}], u'input_transfer_started': None, u'_id': ObjectId('54256ce9d3259d2b071d1657'), u'input_transfer_finished': None, u'nodes': None, u'log': []} 2014:09:26 14:41:02 radical.pilot.MainProcess: [INFO ] Using pilot agent /private/tmp/test/lib/python2.7/site-packages/radical/pilot/agent/radical-pilot-agent-multicore.py 2014:09:26 14:41:02 radical.pilot.MainProcess: [INFO ] Using bootstrapper /private/tmp/test/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Creating agent sandbox 'sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256ce9d3259d2b071d1657/'. 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] saga.fs.Directory ('sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256ce9d3259d2b071d1657/') 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] PTYProcess init <saga.utils.pty_process.PTYProcess object at 0x10a23bd10> 2014:09:26 14:41:02 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshcharlie%h_%p.laughton.ctrl laughton@stampede.tacc.utexas.edu 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] write: [ 9] [ 33](export PS1='$' ; set prompt='$'n) 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] PTYProcess del <saga.utils.pty_process.PTYProcess object at 0x1099eecd0> 2014:09:26 14:41:02 radical.pilot.MainProcess: [INFO ] Starting InputFileTransferWorker 2014:09:26 14:41:02 radical.pilot.MainProcess: [INFO ] Starting InputFileTransferWorker 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Worker thread (ID: Thread-3[4486144000]) for UnitManager 54256ceed3259d2b071d1658 started. 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 54256ceed3259d2b071d1658. 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 54256ceed3259d2b071d1658. 2014:09:26 14:41:02 radical.pilot.MainProcess: [INFO ] Loaded scheduler: DirectSubmissionScheduler. 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 54256ceed3259d2b071d1658. 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 54256ceed3259d2b071d1658. 2014:09:26 14:41:02 radical.pilot.MainProcess: [INFO ] ComputePilot '54256ce9d3259d2b071d1657' state changed from 'PendingLaunch' to 'Launching'. 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.crd > min0.crd' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.crd', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min0.crd', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'penta.top', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/min.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/min.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min.in', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/mdshort.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/mdshort.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'mdshort.in', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string 'md0.ncdf > md_0_0.ncdf' into dict '{'action': 'Transfer', 'source': 'md0.ncdf', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'md_0_0.ncdf', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.crd > min0.crd' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.crd', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min0.crd', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'penta.top', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/min.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/min.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min.in', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/mdshort.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/mdshort.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'mdshort.in', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string 'md0.ncdf > md_0_1.ncdf' into dict '{'action': 'Transfer', 'source': 'md0.ncdf', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'md_0_1.ncdf', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.crd > min0.crd' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.crd', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min0.crd', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'penta.top', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/min.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/min.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min.in', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/mdshort.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/mdshort.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'mdshort.in', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string 'md0.ncdf > md_0_2.ncdf' into dict '{'action': 'Transfer', 'source': 'md0.ncdf', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'md_0_2.ncdf', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.crd > min0.crd' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.crd', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min0.crd', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'penta.top', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/min.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/min.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min.in', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/mdshort.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/mdshort.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'mdshort.in', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string 'md0.ncdf > md_0_3.ncdf' into dict '{'action': 'Transfer', 'source': 'md0.ncdf', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'md_0_3.ncdf', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.crd > min0.crd' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.crd', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min0.crd', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'penta.top', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/min.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/min.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min.in', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/mdshort.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/mdshort.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'mdshort.in', 'priority': 0}' 2014:09:26 14:41:02 radical.pilot.MainProcess: [DEBUG ] Converting string 'md0.ncdf > md_0_4.ncdf' into dict '{'action': 'Transfer', 'source': 'md0.ncdf', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'md_0_4.ncdf', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.crd > min0.crd' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.crd', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min0.crd', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'penta.top', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/min.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/min.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min.in', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/mdshort.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/mdshort.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'mdshort.in', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string 'md0.ncdf > md_0_5.ncdf' into dict '{'action': 'Transfer', 'source': 'md0.ncdf', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'md_0_5.ncdf', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.crd > min0.crd' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.crd', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min0.crd', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'penta.top', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/min.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/min.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min.in', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/mdshort.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/mdshort.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'mdshort.in', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string 'md0.ncdf > md_0_6.ncdf' into dict '{'action': 'Transfer', 'source': 'md0.ncdf', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'md_0_6.ncdf', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.crd > min0.crd' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.crd', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min0.crd', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/penta.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/penta.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'penta.top', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/min.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/min.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'min.in', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/coco_examples/mdshort.in' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/coco_examples/mdshort.in', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'mdshort.in', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] Converting string 'md0.ncdf > md_0_7.ncdf' into dict '{'action': 'Transfer', 'source': 'md0.ncdf', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'md_0_7.ncdf', 'priority': 0}' 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54256ceed3259d2b071d1659' state changed from 'New' to 'PendingInputStaging'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] Scheduled ComputeUnits [] for execution on ComputePilot '54256ce9d3259d2b071d1657'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54256ceed3259d2b071d165b' state changed from 'New' to 'PendingInputStaging'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] 0 units remain unscheduled 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54256ceed3259d2b071d165c' state changed from 'New' to 'PendingInputStaging'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54256cefd3259d2b071d165e' state changed from 'New' to 'PendingInputStaging'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54256cefd3259d2b071d165f' state changed from 'New' to 'PendingInputStaging'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54256cefd3259d2b071d1660' state changed from 'New' to 'PendingInputStaging'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] Creating ComputeUnit sandbox directory sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256ce9d3259d2b071d1657//unit-54256ceed3259d2b071d165a. 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54256ceed3259d2b071d165a' state changed from 'New' to 'PendingInputStaging'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] saga.fs.Directory ('sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256ce9d3259d2b071d1657//unit-54256ceed3259d2b071d165a') 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54256ceed3259d2b071d165d' state changed from 'New' to 'PendingInputStaging'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54256ceed3259d2b071d165a' state changed from 'PendingInputStaging' to 'StagingInput'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] Creating ComputeUnit sandbox directory sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256ce9d3259d2b071d1657//unit-54256ceed3259d2b071d165d. 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] saga.fs.Directory ('sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256ce9d3259d2b071d1657//unit-54256ceed3259d2b071d165d') 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54256ceed3259d2b071d165d' state changed from 'PendingInputStaging' to 'StagingInput'. 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 1023](Last login: Fri Sep 26 08:41:0 ... guides/nUser News: http:) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 1024](//www.tacc.utexas.edu/user-ser ... has three parallel file system)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] write: [ 9] [ 47](stty -echo ; unset HISTFILE ; exec /bin/sh -in) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 350](s: $HOME %28permanent,n quota'd, backed-up%29 $WORK %28permanent, quota'd, not backed-up%29 andn $SCRATCH %28high-speed purged storage%29. The "cdw" and "cds" aliasesn are provided as a convenience to change to your $WORK and $SCRATCHn directories, respectively.n__nn) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] PTYProcess init <saga.utils.pty_process.PTYProcess object at 0x10b6ab5d0> 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshcharlie%h_%p.laughton.ctrl laughton@stampede.tacc.utexas.edu 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] write: [ 14] [ 33](export PS1='$' ; set prompt='$'n) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] read : [ 14] [ 1023](Last login: Fri Sep 26 08:41:0 ... guides/nUser News: http:) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] read : [ 14] [ 1024](//www.tacc.utexas.edu/user-ser ... has three parallel file system)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] write: [ 14] [ 47](stty -echo ; unset HISTFILE ; exec /bin/sh -in) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] PTYProcess init <saga.utils.pty_process.PTYProcess object at 0x10b6ab650> 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] read : [ 14] [ 350](s: $HOME %28permanent,n quota'd, backed-up%29 $WORK %28permanent, quota'd, not backed-up%29 andn $SCRATCH %28high-speed purged storage%29. The "cdw" and "cds" aliasesn are provided as a convenience to change to your $WORK and $SCRATCHn directories, respectively.n__nn) 2014:09:26 14:41:03 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshcharlie%h_%p.laughton.ctrl laughton@stampede.tacc.utexas.edu 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] write: [ 15] [ 33](export PS1='$' ; set prompt='$'n) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 243](--------------------- Project balances for user laughton ----------------------n| Name Avail SUs Expires | |n| TG-MCB090174 676178 2014-09-30 | |n) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] read : [ 15] [ 1023](Last login: Fri Sep 26 08:41:0 ... guides/nUser News: http:) 2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] read : [ 15] [ 1024](//www.tacc.utexas.edu/user-ser ... has three parallel file system)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] write: [ 15] [ 47](stty -echo ; unset HISTFILE ; exec /bin/sh -in)
2014:09:26 14:41:03 radical.pilot.MainProcess: [DEBUG ] read : [ 15] [ 350](s: $HOME %28permanent,n quota'd, backed-up%29 $WORK %28permanent, quota'd, not backed-up%29 andn $SCRATCH %28high-speed purged storage%29. The "cdw" and "cds" aliasesn are provided as a convenience to change to your $WORK and $SCRATCHn directories, respectively.n__nn)
2014:09:26 14:41:04 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 405](------------------------ Disk quotas for user laughton ------------------------n| Disk Usage %28GB%29 Limit %Used File Usage Limit %Used |n| /home1 1.1 5.0 21.18 21903 150000 14.60 |n| /work 223.8 1024.0 21.85 7963 3000000 0.27 |n-------------------------------------------------------------------------------n)
2014:09:26 14:41:04 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 214](nTip 36 %28See "module help tacc_tips" for features or how to disable)\n\n Tired of typing that long directory name? Use tab-completion instead:\n $ cd long
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>) 2014:09:26 14:41:05 radical.pilot.MainProcess: [DEBUG ] write: [ 18] [ 47](stty -echo ; unset HISTFILE ; exec /bin/sh -in) 2014:09:26 14:41:05 radical.pilot.MainProcess: [DEBUG ] read : [ 18] [ 350](s: $HOME %28permanent,n quota'd, backed-up%29 $WORK %28permanent, quota'd, not backed-up%29 andn $SCRATCH %28high-speed purged storage%29. The "cdw" and "cds" aliasesn are provided as a convenience to change to your $WORK and $SCRATCHn directories, respectively.n__nn) 2014:09:26 14:41:05 radical.pilot.MainProcess: [DEBUG ] read : [ 15] 10 2014:09:26 14:41:05 radical.pilot.MainProcess: [DEBUG ] PTYProcess del <saga.utils.pty_process.PTYProcess object at 0x10b6ab5d0> 2014:09:26 14:41:05 radical.pilot.MainProcess: [DEBUG ] PTYProcess del <saga.utils.pty_process.PTYProcess object at 0x10b6ab650> 2014:09:26 14:41:05 radical.pilot.MainProcess: [INFO ] Processing input file transfers for ComputeUnit 54256ceed3259d2b071d165a 2014:09:26 14:41:05 radical.pilot.MainProcess: [DEBUG ] Transferring input file file://localhost//tmp/ExTASY/coco_examples/penta.crd -> sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256ce9d3259d2b071d1657//unit-54256ceed3259d2b071d165a/min0.crd 2014:09:26 14:41:05 radical.pilot.MainProcess: [DEBUG ] saga.fs.File ('file://localhost//tmp/ExTASY/coco_examples/penta.crd') 2014:09:26 14:41:05 radical.pilot.MainProcess: [INFO ] Processing input file transfers for ComputeUnit 54256ceed3259d2b071d165d 2014:09:26 14:41:05 radical.pilot.MainProcess: [DEBUG ] Transferring input file file://localhost//tmp/ExTASY/coco_examples/penta.crd -> sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256ce9d3259d2b071d1657//unit-54256ceed3259d2b071d165d/min0.crd 2014:09:26 14:41:05 radical.pilot.MainProcess: [DEBUG ] saga.fs.File ('file://localhost//tmp/ExTASY/coco_examples/penta.crd') 2014:09:26 14:41:05 radical.pilot.MainProcess: [DEBUG ] read : [ 18] [ 243](--------------------- Project balances for user laughton ----------------------n| Name Avail SUs Expires | |n| TG-MCB090174 676178 2014-09-30 | |n) 2014:09:26 14:41:06 radical.pilot.MainProcess: [DEBUG ] read : [ 18] [ 405](------------------------ Disk quotas for user laughton ------------------------n| Disk Usage %28GB%29 Limit %Used File Usage Limit %Used |n| /home1 1.1 5.0 21.18 21906 150000 14.60 |n| /work 223.8 1024.0 21.85 7963 3000000 0.27 |n-------------------------------------------------------------------------------n) 2014:09:26 14:41:06 radical.pilot.MainProcess: [DEBUG ] read : [ 18] [ 174](nTip 29 %28See "module help tacc_tips" for features or how to disable)\n\n If you want to see just the available default modules execute:\n \n $ module -d avail\n\n) 2014:09:26 14:41:06 radical.pilot.MainProcess: [DEBUG ] read : [ 18] [ 21](login1.stampede%281%29$ $) 2014:09:26 14:41:06 radical.pilot.MainProcess: [DEBUG ] write: [ 18] [ 100](unset PROMPT_COMMAND ; unset HISTFILE ; PS1='PROMPT-$?->'; PS2=''; export PS1 PS2 2>&1 >/dev/nulln) 2014:09:26 14:41:06 radical.pilot.MainProcess: [DEBUG ] read : [ 18] 1 2014:09:26 14:41:06 radical.pilot.MainProcess: [DEBUG ] read : [ 18] 10 2014:09:26 14:41:06 radical.pilot.MainProcess: [DEBUG ] PTYProcess init <saga.utils.pty_process.PTYProcess object at 0x10b6cf0d0> 2014:09:26 14:41:06 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/sftp -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshcharlie%h_%p.laughton.ctrl laughton@stampede.tacc.utexas.edu 2014:09:26 14:41:06 radical.pilot.MainProcess: [DEBUG ] write: [ 14] [ 33](export PS1='$' ; set prompt='$'n) 2014:09:26 14:41:08 radical.pilot.MainProcess: [DEBUG ] read : [ 14] [ 40](Connected to stampede.tacc.utexas.edu.n) 2014:09:26 14:41:08 radical.pilot.MainProcess: [DEBUG ] read : [ 14] [ 30](sftp> Invalid command.nsftp) 2014:09:26 14:41:08 radical.pilot.MainProcess: [DEBUG ] got initial shell prompt (5) (Connected to stampede.tacc.utexas.edu. sftp> Invalid command. sftp> ) 2014:09:26 14:41:08 radical.pilot.MainProcess: [DEBUG ] Got initial shell prompt (5) (Connected to stampede.tacc.utexas.edu. sftp> Invalid command. sftp> ) 2014:09:26 14:41:08 radical.pilot.MainProcess: [DEBUG ] write: [ 14] [ 185](mput "//private/tmp/test/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh" "/home1/01915/laughton/radical.pilot.sandbox/pilot-54256ce9d3259d2b071d1657" nn) 2
CharlieLaughton
commented
10 years ago Hi Vivek,
Here is the output from the Gromacs/LSDMap example.
Charlie
2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] radical.pilot version: 0.19 2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] using database url mongodb://localhost:27017/ 2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] using database name radicalpilot 2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/das4.json 2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/epsrc.json 2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/futuregrid.json 2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/iu.json 2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/localhost.json 2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/lrz.json 2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/ncar.json 2014:09:26 14:51:06 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/test/lib/python2.7/site-packages/radical/pilot/configs/xsede.json 2014:09:26 14:51:07 radical.pilot.MainProcess: [INFO ] New Session created{'database_url': 'mongodb://localhost:27017/', 'database_name': 'radicalpilot', 'last_reconnect': None, 'uid': '54256f4ad3259d2b8d465fa7', 'created': datetime.datetime(2014, 9, 26, 13, 51, 7, 157086)}. 2014:09:26 14:51:07 radical.pilot.MainProcess: [DEBUG ] Worker thread (ID: Thread-1[4406861824]) for PilotManager 54256f4bd3259d2b8d465fa8 started. 2014:09:26 14:51:07 radical.pilot.MainProcess: [DEBUG ] saga.utils.PTYShell ('sftp://stampede.tacc.utexas.edu/') 2014:09:26 14:51:07 radical.pilot.MainProcess: [DEBUG ] PTYShell init <saga.utils.pty_shell.PTYShell object at 0x1062945d0> 2014:09:26 14:51:07 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for PilotManager 54256f4bd3259d2b8d465fa8. 2014:09:26 14:51:07 radical.pilot.MainProcess: [INFO ] PTY prompt pattern: [\$#%>]]\s*$ 2014:09:26 14:51:07 radical.pilot.MainProcess: [DEBUG ] open master pty for [ssh] [laughton@stampede.tacc.utexas.edu] laughton: /usr/bin/env TERM=vt100 "/usr/bin/ssh" -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=yes -o ControlPath=/tmp/saga_sshcharlie%h_%p.laughton.ctrl laughton@stampede.tacc.utexas.edu' 2014:09:26 14:51:07 radical.pilot.MainProcess: [DEBUG ] PTYProcess init <saga.utils.pty_process.PTYProcess object at 0x1062a7d10> 2014:09:26 14:51:07 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=yes -o ControlPath=/tmp/saga_sshcharlie%h%p.laughton.ctrl laughton@stampede.tacc.utexas.edu 2014:09:26 14:51:07 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 33](export PS1='$' ; set prompt='$'n) 2014:09:26 14:51:08 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 51](export PS1='$' > /dev/null 2>&1 || set prompt='$'n) 2014:09:26 14:51:08 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 28](printf 'HELLO%dSAGAn' 1n) 2014:09:26 14:51:09 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 51](export PS1='$' > /dev/null 2>&1 || set prompt='$'n) 2014:09:26 14:51:09 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 28](printf 'HELLO%d_SAGAn' 2n) 2014:09:26 14:51:09 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 1023](Last login: Fri Sep 26 08:35:1 ... guides/nUser News: http:) 2014:09:26 14:51:09 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 1024](//www.tacc.utexas.edu/user-ser ... has three parallel file system)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>) 2014:09:26 14:51:09 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 350](s: $HOME %28permanent,n quota'd, backed-up%29 $WORK %28permanent, quota'd, not backed-up%29 andn $SCRATCH %28high-speed purged storage%29. The "cdw" and "cds" aliasesn are provided as a convenience to change to your $WORK and $SCRATCHn directories, respectively.n__nn) 2014:09:26 14:51:10 radical.pilot.MainProcess: [DEBUG ] write: [ 7] [ 28](printf 'HELLO_%d_SAGAn' 2n) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 243](--------------------- Project balances for user laughton ----------------------n| Name Avail SUs Expires | |n| TG-MCB090174 676178 2014-09-30 | |n) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 405](------------------------ Disk quotas for user laughton ------------------------n| Disk Usage %28GB%29 Limit %Used File Usage Limit %Used |n| /home1 1.1 5.0 21.69 24239 150000 16.16 |n| /work 223.8 1024.0 21.85 7963 3000000 0.27 |n-------------------------------------------------------------------------------n) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 293](nTip 127 %28See "module help tacc_tips" for features or how to disable)\n\n You can put a # at the beginning of a line to save it in the history. This is handy when typing a long command and you need to look something up. You can do this easily in bash with M-# (a.k.a. Esc # or Alt-#).\n\n) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] read : [ 7] [ 68](login4.stampede%281%29$ $$HELLO_1_SAGAn$$HELLO_2_SAGAn$HELLO_2_SAGAn$) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] got shell prompt trigger (4) ( See "man slurm" or the Stampede user guide for more detailed information.
--> To see all the software that is available across all compilers and mpi stacks, issue: "module spider"
--> To see which software packages are available with your currently loaded compiler and mpi stack, issue: "module avail"
--> Stampede has three parallel file systems: $HOME (permanent, quota'd, backed-up) $WORK (permanent, quota'd, not backed-up) and $SCRATCH (high-speed purged storage). The "cdw" and "cds" aliases are provided as a convenience to change to your $WORK and $SCRATCH directories, respectively.
--------------------- Project balances for user laughton ---------------------- | Name Avail SUs Expires | | | TG-MCB090174 676178 2014-09-30 | | ------------------------ Disk quotas for user laughton ------------------------ | Disk Usage (GB) Limit %Used File Usage Limit %Used | | /home1 1.1 5.0 21.69 24239 150000 16.16 |
Tip 127 (See "module help tacc_tips" for features or how to disable)
You can put a # at the beginning of a line to save it in the history. This is handy when typing a long command and you need to look something up. You can do this easily in bash with M-# (a.k.a. Esc # or Alt-#).
login4.stampede(1)$ $$HELLO_1_SAGA $$HELLO_2_SAGA $HELLO_2_SAGA) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] got initial shell prompt (5) ( $) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] Got initial shell prompt (5) ( $) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] PTYProcess init <saga.utils.pty_process.PTYProcess object at 0x1062a7c10> 2014:09:26 14:51:11 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshcharlie%h_%p.laughton.ctrl laughton@stampede.tacc.utexas.edu 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] write: [ 8] [ 33](export PS1='$' ; set prompt='$'n) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 1023](Last login: Fri Sep 26 08:51:0 ... guides/nUser News: http:) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 1024](//www.tacc.utexas.edu/user-ser ... has three parallel file system)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] running command shell: exec /bin/sh -i 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] write: [ 8] [ 47](stty -echo ; unset HISTFILE ; exec /bin/sh -in) 2014:09:26 14:51:11 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 350](s: $HOME %28permanent,n quota'd, backed-up%29 $WORK %28permanent, quota'd, not backed-up%29 andn $SCRATCH %28high-speed purged storage%29. The "cdw" and "cds" aliasesn are provided as a convenience to change to your $WORK and $SCRATCHn directories, respectively.n__nn) 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 243](--------------------- Project balances for user laughton ----------------------n| Name Avail SUs Expires | |n| TG-MCB090174 676178 2014-09-30 | |n) 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 405](------------------------ Disk quotas for user laughton ------------------------n| Disk Usage %28GB%29 Limit %Used File Usage Limit %Used |n| /home1 1.1 5.0 21.69 24239 150000 16.16 |n| /work 223.8 1024.0 21.85 7963 3000000 0.27 |n-------------------------------------------------------------------------------n) 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 319](nTip 40 %28See "module help tacc_tips" for features or how to disable)\n\n Here's a trick to execute the last three commands repeatedly. Press the up-arrow key three times, then type Ctrl-O (the letter not the number). This will execute the displayed command and bring up the next one in list. Repeat as desired.\n\n) 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 21](login4.stampede%281%29$ $) 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] write: [ 8] [ 100](unset PROMPT_COMMAND ; unset HISTFILE ; PS1='PROMPT-$?->'; PS2=''; export PS1 PS2 2>&1 >/dev/nulln) 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] read : [ 8] 1 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] read : [ 8] 10 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] got new shell prompt 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] run_sync: echo "WORKDIR: $PWD" 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] write: [ 8] [ 21](echo "WORKDIR: $PWD"n) 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] read : [ 8] [ 42](WORKDIR: /home1/01915/laughtonnPROMPT-0-) 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] Determined remote working directory for sftp://stampede.tacc.utexas.edu/: '/home1/01915/laughton' 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] PTYShell del <saga.utils.pty_shell.PTYShell object at 0x1062945d0> 2014:09:26 14:51:13 radical.pilot.MainProcess: [INFO ] Launching ComputePilot {u'state': u'PendingLaunch', u'commands': [], u'description': {u'project': u'TG-MCB090174', u'resource': u'stampede.tacc.utexas.edu', u'queue': u'normal', u'sandbox': None, u'cleanup': None, u'pilot_agent_priv': None, u'memory': None, u'cores': 64, u'runtime': 60}, u'sagajobid': None, u'started': None, u'cores_per_node': None, u'output_transfer_started': None, u'finished': None, u'submitted': datetime.datetime(2014, 9, 26, 13, 51, 13, 866000), u'output_transfer_finished': None, u'sandbox': u'sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256f4bd3259d2b8d465fa9/', u'pilotmanager': u'54256f4bd3259d2b8d465fa8', u'unitmanager': None, u'heartbeat': None, u'statehistory': [{u'timestamp': datetime.datetime(2014, 9, 26, 13, 51, 13, 864000), u'state': u'PendingLaunch'}], u'input_transfer_started': None, u'_id': ObjectId('54256f4bd3259d2b8d465fa9'), u'input_transfer_finished': None, u'nodes': None, u'log': []} 2014:09:26 14:51:13 radical.pilot.MainProcess: [INFO ] Using pilot agent /private/tmp/test/lib/python2.7/site-packages/radical/pilot/agent/radical-pilot-agent-multicore.py 2014:09:26 14:51:13 radical.pilot.MainProcess: [INFO ] Using bootstrapper /private/tmp/test/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] Creating agent sandbox 'sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256f4bd3259d2b8d465fa9/'. 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] saga.fs.Directory ('sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256f4bd3259d2b8d465fa9/') 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] PTYProcess init <saga.utils.pty_process.PTYProcess object at 0x106af2d50> 2014:09:26 14:51:13 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshcharlie%h_%p.laughton.ctrl laughton@stampede.tacc.utexas.edu 2014:09:26 14:51:13 radical.pilot.MainProcess: [DEBUG ] write: [ 9] [ 33](export PS1='$' ; set prompt='$'n) 2014:09:26 14:51:14 radical.pilot.MainProcess: [DEBUG ] PTYProcess del <saga.utils.pty_process.PTYProcess object at 0x1062a7c10> 2014:09:26 14:51:14 radical.pilot.MainProcess: [INFO ] Starting InputFileTransferWorker 2014:09:26 14:51:14 radical.pilot.MainProcess: [INFO ] Starting InputFileTransferWorker 2014:09:26 14:51:14 radical.pilot.MainProcess: [DEBUG ] Worker thread (ID: Thread-3[4428419072]) for UnitManager 54256f52d3259d2b8d465faa started. 2014:09:26 14:51:14 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 54256f52d3259d2b8d465faa. 2014:09:26 14:51:14 radical.pilot.MainProcess: [INFO ] Loaded scheduler: DirectSubmissionScheduler. 2014:09:26 14:51:14 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 54256f52d3259d2b8d465faa. 2014:09:26 14:51:14 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 54256f52d3259d2b8d465faa. 2014:09:26 14:51:14 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 54256f52d3259d2b8d465faa. 2014:09:26 14:51:14 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 1023](Last login: Fri Sep 26 08:51:1 ... guides/nUser News: http:) 2014:09:26 14:51:14 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 1024](//www.tacc.utexas.edu/user-ser ... has three parallel file system)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>) 2014:09:26 14:51:14 radical.pilot.MainProcess: [DEBUG ] write: [ 9] [ 47](stty -echo ; unset HISTFILE ; exec /bin/sh -in) 2014:09:26 14:51:14 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 350](s: $HOME %28permanent,n quota'd, backed-up%29 $WORK %28permanent, quota'd, not backed-up%29 andn $SCRATCH %28high-speed purged storage%29. The "cdw" and "cds" aliasesn are provided as a convenience to change to your $WORK and $SCRATCHn directories, respectively.n__nn) 2014:09:26 14:51:14 radical.pilot.MainProcess: [INFO ] ComputePilot '54256f4bd3259d2b8d465fa9' state changed from 'PendingLaunch' to 'Launching'. 2014:09:26 14:51:15 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 243](--------------------- Project balances for user laughton ----------------------n| Name Avail SUs Expires | |n| TG-MCB090174 676178 2014-09-30 | |n) 2014:09:26 14:51:15 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 405](------------------------ Disk quotas for user laughton ------------------------n| Disk Usage %28GB%29 Limit %Used File Usage Limit %Used |n| /home1 1.1 5.0 21.69 24239 150000 16.16 |n| /work 223.8 1024.0 21.85 7963 3000000 0.27 |n-------------------------------------------------------------------------------n) 2014:09:26 14:51:15 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 233](nTip 148 %28See "module help tacc_tips" for features or how to disable)\n\n Linking to TACC packages? You don't have to remember their locations. Just use $TACC_THATPACKAGE_DIR, $TACC_THATPACKAGE_INC, and $TACC_THATPACKAGE_LIB\n\n) 2014:09:26 14:51:15 radical.pilot.MainProcess: [DEBUG ] read : [ 9] [ 21](login4.stampede%281%29$ $) 2014:09:26 14:51:15 radical.pilot.MainProcess: [DEBUG ] write: [ 9] [ 100](unset PROMPT_COMMAND ; unset HISTFILE ; PS1='PROMPT-$?->'; PS2=''; export PS1 PS2 2>&1 >/dev/nulln) 2014:09:26 14:51:15 radical.pilot.MainProcess: [DEBUG ] read : [ 9] 1 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] read : [ 9] 10 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] write: [ 9] [ 169](mkdir -p '/home1/01915/laughton/radical.pilot.sandbox/pilot-54256f4bd3259d2b8d465fa9' ; cd '/home1/01915/laughton/radical.pilot.sandbox/pilot-54256f4bd3259d2b8d465fa9'n) 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] read : [ 9] 10 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] PTYProcess del <saga.utils.pty_process.PTYProcess object at 0x106af2d50> 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] Copying bootstrapper 'file://localhost//private/tmp/test/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh' to agent sandbox (sftp://stampede.tacc.utexas.edu/home1/01915/laughton/radical.pilot.sandbox/pilot-54256f4bd3259d2b8d465fa9/). 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] PTYProcess init <saga.utils.pty_process.PTYProcess object at 0x1093f4f90> 2014:09:26 14:51:16 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/Users/charlie/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshcharlie%h_%p.laughton.ctrl laughton@stampede.tacc.utexas.edu 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] write: [ 17] [ 33](export PS1='$' ; set prompt='$'n) 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] read : [ 17] [ 1023](Last login: Fri Sep 26 08:51:1 ... guides/nUser News: http:) 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] read : [ 17] [ 1024](//www.tacc.utexas.edu/user-ser ... has three parallel file system)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>)
Welcome to the Stampede Supercomputer
Texas Advanced Computing Center, The University of Texas at Austin ** Unauthorized use/access is prohibited. **If you log on to this computer system, you acknowledge your awareness of and concurrence with the UT Austin Acceptable Use Policy. The University will prosecute violators to the full extent of the law.
TACC Usage Policies: http://www.tacc.utexas.edu/user-services/usage-policies/
Questions and Problem Reports:
--> XD Projects: help@xsede.org (email) --> TACC Projects: portal.tacc.utexas.edu (web)
Documentation: http://www.tacc.utexas.edu/user-services/user-guides/ User News: http://www.tacc.utexas.edu/user-services/user-news/
Welcome to Stampede, please read these important system notes:
--> Stampede is currently running the SLURM resource manager to schedule all compute resources. Example SLURM job scripts are available on the system at /share/doc/slurm
To run an interactive shell, issue:
srun -p development -t 0:30:00 -n 32 --pty /bin/bash -l
To submit a batch job, issue: sbatch job.mpi
To show all queued jobs, issue: showq
To kill a queued job, issue: scancel <jobId>) 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] write: [ 17] [ 47](stty -echo ; unset HISTFILE ; exec /bin/sh -in) 2014:09:26 14:51:16 radical.pilot.MainProcess: [DEBUG ] read : [ 17] [ 350](s: $HOME %28permanent,n quota'd, backed-up%29 $WORK %28permanent, quota'd, not backed-up%29 andn $SCRATCH %28high-speed purged storage%29. The "cdw" and "cds" aliasesn are provided as a convenience to change to your $WORK and $SCRATCHn directories, respectively.n__nn) 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start0.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start0.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out0.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out0.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start1.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start1.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out1.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out1.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start2.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start2.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out2.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out2.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start3.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start3.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out3.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out3.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start4.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start4.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out4.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out4.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start5.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start5.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out5.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out5.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start6.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start6.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out6.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out6.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start7.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start7.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out7.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out7.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start8.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start8.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out8.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out8.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start9.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start9.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out9.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out9.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start10.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start10.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out10.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out10.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start11.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start11.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out11.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out11.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start12.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start12.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out12.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out12.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start13.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start13.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out13.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out13.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start14.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start14.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out14.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out14.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start15.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start15.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/topol.top', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'topol.top', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/' into dict '{'action': 'Transfer', 'source': '/', 'flags': ['CreateParents', 'SkipFailed'], 'target': '', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string 'out.gro > out15.gro' into dict '{'action': 'Transfer', 'source': 'out.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'out15.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh > run.sh' into dict '{'action': 'Transfer', 'source': '/private/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/run.sh', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'run.sh', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start16.gro > start.gro' into dict '{'action': 'Transfer', 'source': '/private/tmp/ExTASY/gromacs_lsdmap_example/temp/start16.gro', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'start.gro', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp' into dict '{'action': 'Transfer', 'source': '/tmp/ExTASY/gromacs_lsdmap_example/grompp.mdp', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'grompp.mdp', 'priority': 0}' 2014:09:26 14:51:17 radical.pilot.MainProcess: [DEBUG ] Converting string '/tmp/ExTASY/gromacs_lsdmap_example/topol.top' into dict '{'action': 'Transfer', 'sour
vivek-bala
commented
10 years ago I also tried it on Stampede and I got the following error :-
2014:09:26 16:19:56 radical.pilot.MainProcess: [ERROR ] Pilot launching failed: Couldn't get job id from submitted job! sbatch output:
-----------------------------------------------------------------
Welcome to the Stampede Supercomputer
-----------------------------------------------------------------
--> Verifying valid submit host (login1)...OK
--> Verifying valid jobname...OK
--> Enforcing max jobs per user...OK
--> Verifying availability of your home dir (/home1/02734/vivek91)...OK
--> Verifying availability of your work dir (/work/02734/vivek91)...FAILED
(/tmp/test/local/lib/python2.7/site-packages/saga/adaptors/slurm/slurm_job.py +588 (_job_run) : " sbatch output:\n%s" % out))
Traceback (most recent call last):
File "/tmp/test/local/lib/python2.7/site-packages/radical/pilot/controller/pilot_launcher_worker.py", line 459, in run
pilotjob.run()
File "/tmp/test/local/lib/python2.7/site-packages/saga/job/job.py", line 397, in run
return self._adaptor.run (ttype=ttype)
File "/tmp/test/local/lib/python2.7/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function
return sync_function (self, *args, **kwargs)
File "/tmp/test/local/lib/python2.7/site-packages/saga/adaptors/slurm/slurm_job.py", line 1212, in run
self._id = self.js._job_run (self.jd)
File "/tmp/test/local/lib/python2.7/site-packages/saga/adaptors/slurm/slurm_job.py", line 588, in _job_run
" sbatch output:\n%s" % out)
NoSuccess: Couldn't get job id from submitted job! sbatch output:
-----------------------------------------------------------------
Welcome to the Stampede Supercomputer
-----------------------------------------------------------------
--> Verifying valid submit host (login1)...OK
--> Verifying valid jobname...OK
--> Enforcing max jobs per user...OK
--> Verifying availability of your home dir (/home1/02734/vivek91)...OK
--> Verifying availability of your work dir (/work/02734/vivek91)...FAILED
(/tmp/test/local/lib/python2.7/site-packages/saga/adaptors/slurm/slurm_job.py +588 (_job_run) : " sbatch output:\n%s" % out))It works for me now. Similar case with Jordane. This seemed to be something on the Stampede side. Can you try again Charlie ?
andre-merzky
commented
10 years ago Guys, we have been working on the slurm adaptor lately. Would you mind letting us know what saga-python branch you are using (if you use an explicit branch at all that is)?
Thanks!
vivek-bala
commented
10 years ago No explicit saga-python branch. I think it might be the one on pypi, since its done while installing Radical Pilot.
marksantcroos
commented
10 years ago This seemed to be something on the Stampede side.
"Users may not have been able to access the /work filesystem on Stampede and Maverick between 2:30PM and 3:30PM (CDT) today. Administrators have restored access at this time."
vivek-bala
commented
10 years ago What Mark said is what prompted me to think maybe its on the Stampede side as well since the tool works on Archer as expected. But the message from Stampede was for today and Charlie faced the issue before that.
andre-merzky
commented
10 years ago No explicit saga-python branch. I think it might be the one on pypi, since its done while installing Radical Pilot.
Ah, thank god, then its at least unrelated to those changes. The slurm adaptor has otherwise not been touched for quite a while I think, so I doubt its the (original) source of the problem...
CharlieLaughton
commented
10 years ago I have tried both LSDMAP and COCO examples again this morning, but both still fail. I tried to paste the output here but apparently it's too long. so I will send by email to Vivek.
CharlieLaughton
commented
10 years ago Hi Vivek,
Attached are the log files from this morning’s attempts.
Regards,
Charlie
From: "Vivekanandan Balasubramanian (Vivek)" notifications@github.com<mailto:notifications@github.com> Reply-To: radical-cybertools/ExTASY reply@reply.github.com<mailto:reply@reply.github.com> Date: Friday, 26 September 2014 23:11 To: radical-cybertools/ExTASY ExTASY@noreply.github.com<mailto:ExTASY@noreply.github.com> Cc: CharlieLaughton charles.laughton@nottingham.ac.uk<mailto:charles.laughton@nottingham.ac.uk> Subject: Re: [ExTASY] LSDMap and CoCo workflow examples fail (#51)
It works for me now. Similar case with Jordane. This seemed to be something on the Stampede side. Can you try again Charlie ?
— Reply to this email directly or view it on GitHubhttps://github.com/radical-cybertools/ExTASY/issues/51#issuecomment-57027214.
This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham.
This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.
oleweidner
commented
10 years ago Hi Charlie,
can you please upload the comment as a “gist” at https://gist.github.com/ and add the resulting URLs to the ticket? Otherwise it will become difficult to keep all the strands together.
Many thanks!
Cheers, Ole
On Sep 29, 2014, at 12:55 PM, CharlieLaughton notifications@github.com wrote:
I have tried both LSDMAP and COCO examples again this morning, but both still fail. I tried to paste the output here but apparently it's too long. so I will send by email to Vivek.
— Reply to this email directly or view it on GitHub.
CharlieLaughton
commented
10 years ago oleweidner
commented
10 years ago Cheers ;-)
vivek-bala
commented
10 years ago This is in Stampede right ? This might be the case of no space on $HOME. Could you try it again after deleting unwanted files/folders in $HOME (radical.pilot.sandbox). Also, the gist file seems to have some html tags as well. Is it possible to paste just the RP log ?
oleweidner
commented
10 years ago Vivek. Please set the default sandbox for stampede to $SCRATCH. $HOME is not useful for any serious computations.
oleweidner
commented
10 years ago ... $WORK that is.
vivek-bala
commented
10 years ago On Stampede, there is a number following each username in $WORK or $HOME, not sure how to put up a default sandbox in this case.
marksantcroos
commented
10 years ago On 01 Oct 2014, at 10:22 , Vivekanandan (Vivek) Balasubramanian notifications@github.com wrote:
On Stampede, there is a number following each username in $WORK or $HOME, not sure how to put up a default sandbox in this case.
AFAIK this is a RP concern and not an ExTASY concern.
For RP there is already a ticket: https://github.com/radical-cybertools/radical.pilot/issues/394
I will take care of that, if you could test it afterwards, that would be great.=
vivek-bala
commented
10 years ago Sure.
CharlieLaughton
commented
10 years ago I am testing the latest (as at 6th October) version of the instructions at https://github.com/radical-cybertools/ExTASY/blob/devel/README.md, I install ExTASY on my Mac and CoCo on Stampede OK. But on running I get:
[ExTASY-tools] [d-156-137-4:~/coam-on-stampede] charlie% extasy --RPconfig archer.rcfg --Kconfig cocoamber.wcfg
2014:10:06 12:45:28 radical.pilot.MainProcess: [INFO ] radical.pilot version: 0.20
2014:10:06 12:45:28 radical.pilot.MainProcess: [INFO ] using database url mongodb://ec2-184-72-89-141.compute-1.amazonaws.com:27017/
2014:10:06 12:45:28 radical.pilot.MainProcess: [INFO ] using database name radicalpilot
2014:10:06 12:45:28 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ExTASY-tools/lib/python2.7/site-packages/radical/pilot/configs/das4.json
2014:10:06 12:45:28 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ExTASY-tools/lib/python2.7/site-packages/radical/pilot/configs/epsrc.json
2014:10:06 12:45:28 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ExTASY-tools/lib/python2.7/site-packages/radical/pilot/configs/futuregrid.json
2014:10:06 12:45:29 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ExTASY-tools/lib/python2.7/site-packages/radical/pilot/configs/iu.json
2014:10:06 12:45:29 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ExTASY-tools/lib/python2.7/site-packages/radical/pilot/configs/localhost.json
2014:10:06 12:45:29 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ExTASY-tools/lib/python2.7/site-packages/radical/pilot/configs/lrz.json
2014:10:06 12:45:29 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ExTASY-tools/lib/python2.7/site-packages/radical/pilot/configs/ncar.json
2014:10:06 12:45:29 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ExTASY-tools/lib/python2.7/site-packages/radical/pilot/configs/radical.json
2014:10:06 12:45:29 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ExTASY-tools/lib/python2.7/site-packages/radical/pilot/configs/xsede.json
Traceback (most recent call last):
File "/tmp/ExTASY-tools/bin/extasy", line 9, in
radical.pilot.exceptions.PilotException: Couldn't create new session (database URL 'mongodb://ec2-184-72-89-141.compute-1.amazonaws.com:27017/' incorrect?): timed out
I get the same message if I try to run the workflow on Archer.
CharlieLaughton
commented
10 years ago Update: the error message suggests that, for some reason, there is a problem contacting the mongo DB server. I therefore ran a local server and replaced 'ec2-...' etc. with 'localhost'. The coco-amber run now starts promisingly but eventually fails. The tail of the log message is:
[...]
[Callback]: ComputeUnit '54328a6f9e3d0a54d23188f1' state changed to PendingExecution.
2014:10:06 13:27:47 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '54328a6f9e3d0a54d23188f4' state changed from 'StagingInput' to 'PendingExecution'.
[Callback]: ComputeUnit '54328a6f9e3d0a54d23188f4' state changed to PendingExecution.
2014:10:06 13:28:14 radical.pilot.MainProcess: [INFO ] Performing periodical health check for 54328a669e3d0a54d23188ed (SAGA job id [slurm+ssh://stampede.tacc.utexas.edu]-[4227650])
2014:10:06 13:28:14 radical.pilot.MainProcess: [DEBUG ] write: [ 27] [ 26](scontrol show job 4227650n)
2014:10:06 13:28:15 radical.pilot.MainProcess: [DEBUG ] read : [ 27] [ 967](JobId=4227650 Name=SAGAPythonS ... 9e3d0a54d23188ednnPROMPT-0-)
2014:10:06 13:28:15 radical.pilot.MainProcess: [ERROR ] SAGA job state for ComputePilot 54328a669e3d0a54d23188ed is Failed.
2014:10:06 13:28:15 radical.pilot.MainProcess: [ERROR ] pilot 54328a669e3d0a54d23188ed declared dead
2014:10:06 13:28:15 radical.pilot.MainProcess: [INFO ] ComputePilot '54328a669e3d0a54d23188ed' state changed from 'PendingActive' to 'Failed'.
[Callback]: ComputePilot '54328a669e3d0a54d23188ed' state changed to Failed.
pilot manager controller thread caught system exit -- forcing application shutdown
Traceback (most recent call last):
File "/tmp/ExTASY-tools/bin/extasy", line 9, in
vivek-bala
commented
10 years ago Where is the mongodb instance being run ? Both the local machine and the remote host would need public access to the mongodb server instance. I'm not sure why the amazon instance would not get connected. Looking into it.
CharlieLaughton
commented
10 years ago The mongodb instance was being run on my Mac - and so was almost certainly not visible to Stampede, hence the new set of error messages, so the immediate above is probably largely irrelevant. Working from home I have no mongodb problems with the 'standard' setup so the problem is probably something weird at Nottingham with firewalls or something.
There are still problems though. Cycle 0 runs fine - the eight MD jobs run no problem, and the trajectory files are staged back. The CoCo job also runs fine. But the second cycle of MD (Cycle 1) hangs while staging output. My system monitor tells me that python is downloading plenty of bytes, but no new files, or file stubs. appear.
[...]
Cycle 1
Starting Simulation
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8b' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b89' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8e' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8a' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8f' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b90' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8d' state changed to PendingInpuStaging.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8c' state changed to PendingInpuStaging.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b89' state changed to StagingInpu.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8b' state changed to StagingInpu.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8a' state changed to StagingInpu.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b89' state changed to PendingExection.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8e' state changed to StagingInpu.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b89' state changed to Scheduling.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b89' state changed to Executing.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b90' state changed to StagingInpu.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8b' state changed to PendingExection.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b89' state changed to StagingOutpt.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8b' state changed to Executing.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8f' state changed to StagingInpu.
[Callback]: ComputeUnit '5432ca3202529a041a2a1b8b' state changed to StagingOutpt.
(_Output hangs here for about 1 hour_)
[Callback]: ComputePilot '5432c97902529a041a2a1b7e' state changed to Failed.
pilot manager controller thread caught system exit -- forcing application shutdown
Traceback (most recent call last):
File "/tmp/ExTASY-tools/bin/extasy", line 9, in
oleweidner
commented
10 years ago Hi Charlie, I have updated the documentation for MongoDB and SSH set up for the next iteration of the EnsembleMD toolkit. Would you mind having a quick look at it an tell me whether it makes things a bit clearer (specifically the MongoDB section) ?
http://radicalensemblemd.readthedocs.org/en/latest/installation.html#preparing-the-environment
CharlieLaughton
commented
10 years ago Hi Ole, the documentation looks good. However I'm not clear on how that relates to this ticket? I have retried the Mac-Stampede-Amber-CoCo case again and nothing has changed - the workflow hangs at the same point as before (which appears to be the same point found by Ardita for the Linux-Stampede-Amber-CoCo scenario).
CharlieLaughton
commented
10 years ago Update: I have just been able to more-or-less successfully run the Amber-CoCo workflow on Stampede from testing.saga-project.org. Both MD/analysis cycles complete and output files stage back, however the job does not die gracefully:
[...] [Callback]: ComputeUnit '54395f3520a64112e5fa9934' state changed to Done. Analysis Execution time : 27.086 [Callback]: ComputePilot '54395e5a20a64112e5fa9921' state changed to Canceled. Exception in thread InputFileTransferWorker-1 (most likely raised during interpreter shutdown):Exception in thread OutputFileTransferWorker-1 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
File "/tmp/ExTASY-tools/local/lib/python2.7/site-packages/radical/pilot/controller/output_file_transfer_worker.py", line 199, in runTraceback (most recent call last):
File "/tmp/ExTASY-tools/local/lib/python2.7/site-packages/pymongo/collection.py", line 877, in find File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner <type 'exceptions.TypeError'>: 'NoneType' object is not callable File "/tmp/ExTASY-tools/local/lib/python2.7/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 194, in run
File "/tmp/ExTASY-tools/local/lib/python2.7/site-packages/pymongo/collection.py", line 877, in find <type 'exceptions.TypeError'>: 'NoneType' object is not callable (ExTASY-tools)extasy-project@ip-10-243-91-117:~/coam-on-stampede$
andre-merzky
commented
10 years ago Note: the shutdown issue is know (see https://github.com/radical-cybertools/radical.pilot/issues/310).
CharlieLaughton
commented
10 years ago Update: back to the Mac-Stampede-Amber-CoCo combination after the "successful" run from test.ing.sga-project.org, the job still fails, but not almost immediately as it was a couple of days ago. Now (after a fresh install of everything) we are back to the situation we had before:
It seems to me that this pattern suggests the problem is somewhere in the file transfers, that attempts to simultaneously stage in the data for one compute unit while staging out the data for another just locks up. Maybe it didn't happen when the jobs were run from testing.saga-project.org because file uploads all completed before any downloads were started, maybe due to a higher bandwidth from there to stampede. Is this possible?
andre-merzky
commented
10 years ago Hi Charlie,
we also suspect that the culprit of the repeated problems is the file transfer layer, or our use of that layer. Currently we suspect the problem to be related to the time duration over which that channel is kept open, but we also investigate in other directions (size,number and frequency of transfers, configurations of participating hosts, etc).
oleweidner
commented
10 years ago related to #82
oleweidner
commented
10 years ago Not reproducible. Closed before new round of testing.
Testing the Mac/Stampede combination, neither Gromacs/LSDMap nor AMBER/CoCo workflows run to completion. Both fail with very similar error messages:
test] [d-156-137-4:/tmp/ExTASY] charlie% extasy --RPconfig /tmp/ExTASY/config/RP_config.py --Kconfig /tmp/ExTASY/config/gromacs_lsdmap_config.py Session UID: 5425493cd3259d28034e0c9d Pilot UID : 5425493fd3259d28034e0c9f Loading kernel configurations from /private/tmp/test/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/amber.json Loading kernel configurations from /private/tmp/test/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/coco.json Loading kernel configurations from /private/tmp/test/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/gromacs.json Loading kernel configurations from /private/tmp/test/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/lsdmap.json Loading kernel configurations from /private/tmp/test/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/mmpbsa.json Loading kernel configurations from /private/tmp/test/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/namd.json Loading kernel configurations from /private/tmp/test/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/sleep.json Loading kernel configurations from /private/tmp/test/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/test.json Prepare grofiles.. [Callback]: ComputePilot '5425493fd3259d28034e0c9f' state changed to Launching. Preprocessing time : 2.60613107681 Cycle 0 Starting Simulation [Callback]: ComputeUnit '5425494ad3259d28034e0ca2' state changed to PendingInputStaging. [Callback]: ComputeUnit '5425494ad3259d28034e0ca3' state changed to PendingInputStaging.
[…]
[Callback]: ComputeUnit '5425494ad3259d28034e0ccb' state changed to StagingInput. [Callback]: ComputeUnit '5425494ad3259d28034e0cb7' state changed to StagingInput. [Callback]: ComputeUnit '5425494ad3259d28034e0ca2' state changed to PendingExecution. [Callback]: ComputeUnit '5425494ad3259d28034e0cbc' state changed to StagingInput. [Callback]: ComputeUnit '5425494ad3259d28034e0cc5' state changed to StagingInput. [Callback]: ComputeUnit '5425494ad3259d28034e0cc9' state changed to PendingExecution. [Callback]: ComputeUnit '5425494ad3259d28034e0ca4' state changed to StagingInput. [Callback]: ComputePilot '5425493fd3259d28034e0c9f' state changed to Failed. pilot manager controller thread caught system exit -- forcing application shutdown Traceback (most recent call last): File "/tmp/test/bin/extasy", line 9, in
load_entry_point('radical.ensemblemd.extasy==0.1', 'console_scripts', 'extasy')()
File "/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/runme.py", line 130, in main
Simulator(umgr,RPconfig,Kconfig,i)
File "/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Gromacs/simulator.py", line 51, in Simulator
umgr.wait_units()
File "/tmp/test/lib/python2.7/site-packages/radical/pilot/unit_manager.py", line 669, in wait_units
time.sleep (0.1)
KeyboardInterrupt
[test] [d-156-137-4:/tmp/ExTASY] charlie%
The Amber/Coco option ends similarly:
[…] [Callback]: ComputeUnit '54254c27d3259d28a296ebae' state changed to StagingInput. [Callback]: ComputeUnit '54254c27d3259d28a296ebaa' state changed to StagingInput. [Callback]: ComputeUnit '54254c28d3259d28a296ebaf' state changed to PendingExecution. [Callback]: ComputeUnit '54254c27d3259d28a296ebac' state changed to StagingInput. [Callback]: ComputeUnit '54254c28d3259d28a296ebb1' state changed to StagingInput. [Callback]: ComputeUnit '54254c27d3259d28a296ebab' state changed to PendingExecution. [Callback]: ComputeUnit '54254c27d3259d28a296ebae' state changed to PendingExecution. [Callback]: ComputeUnit '54254c27d3259d28a296ebaa' state changed to PendingExecution. [Callback]: ComputeUnit '54254c27d3259d28a296ebac' state changed to PendingExecution. [Callback]: ComputeUnit '54254c28d3259d28a296ebb1' state changed to PendingExecution. [Callback]: ComputePilot '54254c21d3259d28a296eba8' state changed to Failed. pilot manager controller thread caught system exit -- forcing application shutdown Traceback (most recent call last): File "/tmp/test/bin/extasy", line 9, in
load_entry_point('radical.ensemblemd.extasy==0.1', 'console_scripts', 'extasy')()
File "/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/runme.py", line 130, in main
Simulator(umgr,RPconfig,Kconfig,i)
File "/tmp/test/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Amber/simulator.py", line 55, in Simulator
umgr.wait_units()
File "/tmp/test/lib/python2.7/site-packages/radical/pilot/unit_manager.py", line 669, in wait_units
time.sleep (0.1)
KeyboardInterrupt
[test] [d-156-137-4:/tmp/ExTASY/coco_examples] charlie%