CoCo-based workflow of ExTASY on Archer fails

The debugging information and errors (computational units change their state to failed) are the following: (ard)extasy-project@ip-10-243-91-117:/tmp/ExTASY$ RADICAL_PILOT_VERBOSE=info extasy --RPconfig /tmp/ExTASY/config/RP_archer_config.py --Kconfig /tmp/ExTASY/config/amber_coco_config.py 2014:08:25 17:50:37 radical.pilot.MainProcess: [INFO ] radical.pilot version: 0.18 2014:08:25 17:50:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ard/local/lib/python2.7/site-packages/radical/pilot/configs/lrz.json 2014:08:25 17:50:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ard/local/lib/python2.7/site-packages/radical/pilot/configs/epsrc.json 2014:08:25 17:50:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ard/local/lib/python2.7/site-packages/radical/pilot/configs/iu.json 2014:08:25 17:50:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ard/local/lib/python2.7/site-packages/radical/pilot/configs/das4.json 2014:08:25 17:50:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ard/local/lib/python2.7/site-packages/radical/pilot/configs/futuregrid.json 2014:08:25 17:50:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ard/local/lib/python2.7/site-packages/radical/pilot/configs/xsede.json 2014:08:25 17:50:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ard/local/lib/python2.7/site-packages/radical/pilot/configs/ncar.json 2014:08:25 17:50:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /tmp/ard/local/lib/python2.7/site-packages/radical/pilot/configs/localhost.json 2014:08:25 17:50:39 radical.pilot.MainProcess: [INFO ] New Session created{'database_url': 'mongodb://ec2-184-72-89-141.compute-1.amazonaws.com:27017/', 'database_name': 'radicalpilot', 'last_reconnect': None, 'uid': '53fb776e20a6413ee4e157cd', 'created': datetime.datetime(2014, 8, 25, 17, 50, 38, 971359)}. Session UID: 53fb776e20a6413ee4e157cd 2014:08:25 17:50:39 radical.pilot.MainProcess: [INFO ] PTY prompt pattern: [\$#%>]]\s*$ 2014:08:25 17:50:39 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=yes -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 17:50:41 radical.pilot.MainProcess: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk Pilot UID : 53fb776f20a6413ee4e157cf 2014:08:25 17:50:42 radical.pilot.MainProcess: [INFO ] Loaded scheduler: DirectSubmissionScheduler. Cycle : 0 Creating initial setup 2014:08:25 17:50:42 radical.pilot.MainProcess: [INFO ] Scheduled ComputeUnits [] for execution on ComputePilot '53fb776f20a6413ee4e157cf'. 2014:08:25 17:50:43 radical.pilot.PilotLauncherWorker-1: [INFO ] Launching ComputePilot {u'state': u'PendingLaunch', u'commands': [], u'description': {u'project': u'e290', u'resource': u'archer.ac.uk', u'queue': None, u'sandbox': None, u'cleanup': None, u'pilot_agent_priv': None, u'memory': None, u'cores': 64, u'runtime': 60}, u'sagajobid': None, u'started': None, u'cores_per_node': None, u'output_transfer_started': None, u'sandbox': u'sftp://login.archer.ac.uk/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/', u'submitted': datetime.datetime(2014, 8, 25, 17, 50, 42, 335000), u'output_transfer_finished': None, u'finished': None, u'pilotmanager': u'53fb776f20a6413ee4e157ce', u'unitmanager': u'53fb777220a6413ee4e157d0', u'statehistory': [{u'timestamp': datetime.datetime(2014, 8, 25, 17, 50, 42, 334000), u'state': u'PendingLaunch'}], u'wu_queue': [], u'heartbeat': None, u'input_transfer_started': None, u'_id': ObjectId('53fb776f20a6413ee4e157cf'), u'input_transfer_finished': None, u'nodes': None, u'log': []} 2014:08:25 17:50:43 radical.pilot.PilotLauncherWorker-1: [INFO ] Using pilot agent /tmp/ard/lib/python2.7/site-packages/radical/pilot/agent/radical-pilot-agent-multicore.py 2014:08:25 17:50:43 radical.pilot.PilotLauncherWorker-1: [INFO ] Using bootstrapper /tmp/ard/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh 2014:08:25 17:50:43 radical.pilot.InputFileTransferWorker-1: [INFO ] Creating ComputeUnit sandbox directory sftp://login.archer.ac.uk/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb777220a6413ee4e157d1. 2014:08:25 17:50:43 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb777220a6413ee4e157d1' state changed from 'New' to 'TransferringInput'. [Callback]: ComputeUnit '53fb777220a6413ee4e157d1' state changed to TransferringInput. 2014:08:25 17:50:43 radical.pilot.InputFileTransferWorker-1: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=yes -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 17:50:43 radical.pilot.InputFileTransferWorker-1: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 17:50:44 radical.pilot.MainProcess: [INFO ] ComputePilot '53fb776f20a6413ee4e157cf' state changed from 'PendingLaunch' to 'Launching'. 2014:08:25 17:50:44 radical.pilot.MainProcess: [ERROR ] Couldn't call callback function 'NoneType' object has no attribute 'uid' 2014:08:25 17:50:46 radical.pilot.InputFileTransferWorker-1: [INFO ] Processing input file transfers for ComputeUnit 53fb777220a6413ee4e157d1 2014:08:25 17:50:48 radical.pilot.InputFileTransferWorker-1: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 17:50:49 radical.pilot.InputFileTransferWorker-1: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/sftp -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 17:50:53 radical.pilot.InputFileTransferWorker-1: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 17:50:56 radical.pilot.InputFileTransferWorker-1: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 17:51:00 radical.pilot.InputFileTransferWorker-1: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 17:51:04 radical.pilot.InputFileTransferWorker-1: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 17:51:06 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb777220a6413ee4e157d1' state changed from 'TransferringInput' to 'PendingExecution'. [Callback]: ComputeUnit '53fb777220a6413ee4e157d1' state changed to PendingExecution. 2014:08:25 17:51:21 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 17:51:22 radical.pilot.MainProcess: [INFO ] ComputePilot '53fb776f20a6413ee4e157cf' state changed from 'Launching' to 'PendingActive'. 2014:08:25 17:51:22 radical.pilot.MainProcess: [ERROR ] Couldn't call callback function 'NoneType' object has no attribute 'uid' 2014:08:25 17:52:15 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 17:53:10 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 17:54:01 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 17:54:53 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 17:55:47 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 17:56:41 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 17:57:36 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 17:58:30 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 17:59:24 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 18:00:19 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 18:01:13 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 18:02:05 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 18:02:59 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 18:03:50 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 18:04:45 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 18:04:48 radical.pilot.MainProcess: [INFO ] ComputePilot '53fb776f20a6413ee4e157cf' state changed from 'PendingActive' to 'Active'. 2014:08:25 18:04:48 radical.pilot.MainProcess: [ERROR ] Couldn't call callback function 'NoneType' object has no attribute 'uid' 2014:08:25 18:04:49 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb777220a6413ee4e157d1' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb777220a6413ee4e157d1' state changed to Scheduling. 2014:08:25 18:04:50 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb777220a6413ee4e157d1' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb777220a6413ee4e157d1' state changed to Executing. 2014:08:25 18:04:53 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb777220a6413ee4e157d1' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb777220a6413ee4e157d1' state changed to Failed. Log: Transferring input file file://localhost//tmp/ard/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Preprocessor/Amber/cocoUi.py -> sftp://login.archer.ac.uk/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb777220a6413ee4e157d1 Starting Simulation Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit 2014:08:25 18:04:53 radical.pilot.MainProcess: [INFO ] Scheduled ComputeUnits ['53fb7ac520a6413ee4e157d2', '53fb7ac520a6413ee4e157d3', '53fb7ac520a6413ee4e157d4', '53fb7ac520a6413ee4e157d5', '53fb7ac520a6413ee4e157d6', '53fb7ac520a6413ee4e157d7', '53fb7ac520a6413ee4e157d8', '53fb7ac520a6413ee4e157d9'] for execution on ComputePilot '53fb776f20a6413ee4e157cf'. 2014:08:25 18:04:54 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d2' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d2' state changed to PendingExecution. 2014:08:25 18:04:54 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d3' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d3' state changed to PendingExecution. 2014:08:25 18:04:54 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d4' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d4' state changed to PendingExecution. 2014:08:25 18:04:54 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d5' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d5' state changed to PendingExecution. 2014:08:25 18:04:54 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d6' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d6' state changed to PendingExecution. 2014:08:25 18:04:54 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d7' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d7' state changed to PendingExecution. 2014:08:25 18:04:54 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d8' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d8' state changed to PendingExecution. 2014:08:25 18:04:54 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d9' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d9' state changed to PendingExecution. 2014:08:25 18:04:55 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d2' state changed from 'PendingExecution' to 'Executing'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d2' state changed to Executing. 2014:08:25 18:04:56 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d3' state changed from 'PendingExecution' to 'Executing'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d3' state changed to Executing. 2014:08:25 18:04:57 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d2' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d2' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:04:57 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d4' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d4' state changed to Scheduling. 2014:08:25 18:04:58 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d4' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d4' state changed to Executing. 2014:08:25 18:04:58 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d5' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d5' state changed to Scheduling. 2014:08:25 18:04:59 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d3' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d3' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:04:59 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d5' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d5' state changed to Executing. 2014:08:25 18:04:59 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d6' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d6' state changed to Scheduling. 2014:08:25 18:05:00 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d6' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d6' state changed to Executing. 2014:08:25 18:05:01 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d7' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d7' state changed to Scheduling. 2014:08:25 18:05:02 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d4' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d4' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:02 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d5' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d5' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:02 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d6' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d6' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:02 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d7' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d7' state changed to Executing. 2014:08:25 18:05:02 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d8' state changed from 'PendingExecution' to 'Executing'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d8' state changed to Executing. 2014:08:25 18:05:03 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d9' state changed from 'PendingExecution' to 'Executing'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d9' state changed to Executing. 2014:08:25 18:05:05 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d7' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d7' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:05 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d8' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d8' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:05 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ac520a6413ee4e157d9' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ac520a6413ee4e157d9' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. Total Simulation Time : 13.0849468708 Simulation Execution Time : 10.506 Starting Analysis Cycle : 0 Submitting COCO Compute Unit 2014:08:25 18:05:06 radical.pilot.MainProcess: [INFO ] Scheduled ComputeUnits [] for execution on ComputePilot '53fb776f20a6413ee4e157cf'. 2014:08:25 18:05:07 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ad220a6413ee4e157da' state changed from 'New' to 'PendingInputTransfer'. [Callback]: ComputeUnit '53fb7ad220a6413ee4e157da' state changed to PendingInputTransfer. 2014:08:25 18:05:07 radical.pilot.InputFileTransferWorker-2: [INFO ] Creating ComputeUnit sandbox directory sftp://login.archer.ac.uk/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ad220a6413ee4e157da. 2014:08:25 18:05:07 radical.pilot.InputFileTransferWorker-2: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=yes -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 18:05:07 radical.pilot.InputFileTransferWorker-2: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 18:05:08 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ad220a6413ee4e157da' state changed from 'PendingInputTransfer' to 'TransferringInput'. [Callback]: ComputeUnit '53fb7ad220a6413ee4e157da' state changed to TransferringInput. 2014:08:25 18:05:10 radical.pilot.InputFileTransferWorker-2: [INFO ] Processing input file transfers for ComputeUnit 53fb7ad220a6413ee4e157da 2014:08:25 18:05:11 radical.pilot.InputFileTransferWorker-2: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 18:05:12 radical.pilot.InputFileTransferWorker-2: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/sftp -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 18:05:15 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ad220a6413ee4e157da' state changed from 'TransferringInput' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ad220a6413ee4e157da' state changed to Scheduling. 2014:08:25 18:05:16 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ad220a6413ee4e157da' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ad220a6413ee4e157da' state changed to Executing. 2014:08:25 18:05:18 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ad220a6413ee4e157da' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ad220a6413ee4e157da' state changed to Failed. Log: Transferring input file file://localhost//tmp/ard/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Analyzer/CoCo/postexec.py -> sftp://login.archer.ac.uk/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ad220a6413ee4e157da Analysis Execution time : 2.792 Starting Simulation Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit Submitting new 'pmemd' compute unit 2014:08:25 18:05:18 radical.pilot.MainProcess: [INFO ] Scheduled ComputeUnits ['53fb7ade20a6413ee4e157db', '53fb7ade20a6413ee4e157dc', '53fb7ade20a6413ee4e157dd', '53fb7ade20a6413ee4e157de', '53fb7ade20a6413ee4e157df', '53fb7ade20a6413ee4e157e0', '53fb7ade20a6413ee4e157e1', '53fb7ade20a6413ee4e157e2'] for execution on ComputePilot '53fb776f20a6413ee4e157cf'. 2014:08:25 18:05:19 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e2' state changed from 'New' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e2' state changed to Scheduling. 2014:08:25 18:05:19 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157db' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157db' state changed to PendingExecution. 2014:08:25 18:05:19 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157dc' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157dc' state changed to PendingExecution. 2014:08:25 18:05:19 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157dd' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157dd' state changed to PendingExecution. 2014:08:25 18:05:19 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157de' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157de' state changed to PendingExecution. 2014:08:25 18:05:19 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157df' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157df' state changed to PendingExecution. 2014:08:25 18:05:19 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e0' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e0' state changed to PendingExecution. 2014:08:25 18:05:19 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e1' state changed from 'New' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e1' state changed to PendingExecution. 2014:08:25 18:05:21 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e2' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e2' state changed to Executing. 2014:08:25 18:05:21 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157db' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157db' state changed to Scheduling. 2014:08:25 18:05:22 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157db' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157db' state changed to Executing. 2014:08:25 18:05:22 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157dc' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157dc' state changed to Scheduling. 2014:08:25 18:05:23 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e2' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e2' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:23 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157dc' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157dc' state changed to Executing. 2014:08:25 18:05:23 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157dd' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157dd' state changed to Scheduling. 2014:08:25 18:05:24 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157dd' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157dd' state changed to Executing. 2014:08:25 18:05:24 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157de' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157de' state changed to Scheduling. 2014:08:25 18:05:25 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157df' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157df' state changed to Scheduling. 2014:08:25 18:05:26 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157db' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157db' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:26 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157dc' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157dc' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:26 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157dd' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157dd' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:26 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157de' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157de' state changed to Executing. 2014:08:25 18:05:26 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157df' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157df' state changed to Executing. 2014:08:25 18:05:26 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e0' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e0' state changed to Scheduling. 2014:08:25 18:05:27 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e0' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e0' state changed to Executing. 2014:08:25 18:05:28 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e1' state changed from 'PendingExecution' to 'Scheduling'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e1' state changed to Scheduling. 2014:08:25 18:05:29 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157de' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157de' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:29 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157df' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157df' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:29 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e0' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e0' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. 2014:08:25 18:05:30 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e1' state changed from 'Scheduling' to 'Executing'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e1' state changed to Executing. 2014:08:25 18:05:32 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7ade20a6413ee4e157e1' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7ade20a6413ee4e157e1' state changed to Failed. Log: Scheduled for execution on ComputePilot 53fb776f20a6413ee4e157cf. Total Simulation Time : 15.0607271194 Simulation Execution Time : 11.832 Starting Analysis Cycle : 1 Submitting COCO Compute Unit 2014:08:25 18:05:33 radical.pilot.MainProcess: [INFO ] Scheduled ComputeUnits [] for execution on ComputePilot '53fb776f20a6413ee4e157cf'. 2014:08:25 18:05:34 radical.pilot.InputFileTransferWorker-2: [INFO ] Creating ComputeUnit sandbox directory sftp://login.archer.ac.uk/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7aed20a6413ee4e157e3. 2014:08:25 18:05:34 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7aed20a6413ee4e157e3' state changed from 'New' to 'TransferringInput'. [Callback]: ComputeUnit '53fb7aed20a6413ee4e157e3' state changed to TransferringInput. 2014:08:25 18:05:34 radical.pilot.InputFileTransferWorker-2: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 18:05:36 radical.pilot.InputFileTransferWorker-2: [INFO ] Processing input file transfers for ComputeUnit 53fb7aed20a6413ee4e157e3 2014:08:25 18:05:38 radical.pilot.InputFileTransferWorker-2: [INFO ] running: /usr/bin/env TERM=vt100 /usr/bin/ssh -t -o IdentityFile=/home/extasy-project/.ssh/id_rsa -o ControlMaster=no -o ControlPath=/tmp/saga_sshextasy-project%h_%p.ardi.ctrl ardi@login.archer.ac.uk 2014:08:25 18:05:39 radical.pilot.PilotLauncherWorker-1: [INFO ] Performing periodical health check for 53fb776f20a6413ee4e157cf (SAGA job id [pbs+ssh://login.archer.ac.uk]-[509101]) 2014:08:25 18:05:40 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7aed20a6413ee4e157e3' state changed from 'TransferringInput' to 'PendingExecution'. [Callback]: ComputeUnit '53fb7aed20a6413ee4e157e3' state changed to PendingExecution. 2014:08:25 18:05:41 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7aed20a6413ee4e157e3' state changed from 'PendingExecution' to 'Executing'. [Callback]: ComputeUnit '53fb7aed20a6413ee4e157e3' state changed to Executing. 2014:08:25 18:05:44 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '53fb7aed20a6413ee4e157e3' state changed from 'Executing' to 'Failed'. [Callback]: ComputeUnit '53fb7aed20a6413ee4e157e3' state changed to Failed. Log: Transferring input file file://localhost//tmp/ard/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Analyzer/CoCo/postexec.py -> sftp://login.archer.ac.uk/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7aed20a6413ee4e157e3 Analysis Execution time : 2.703 2014:08:25 18:05:44 radical.pilot.MainProcess: [INFO ] Sent 'COMMAND_CANCEL_PILOT' command to all pilots. 2014:08:25 18:05:45 radical.pilot.MainProcess: [INFO ] Closed PilotManager 53fb776f20a6413ee4e157ce. 2014:08:25 18:05:45 radical.pilot.MainProcess: [INFO ] Closed UnitManager 53fb777220a6413ee4e157d0. 2014:08:25 18:05:45 radical.pilot.MainProcess: [INFO ] Deleted session 53fb776e20a6413ee4e157cd from database. 2014:08:25 18:05:45 radical.pilot.MainProcess: [INFO ] Closed Session None.

AGENT.LOG

ardi@eslogin001:/work/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf> more AGENT.LOG 2014-08-25 19:04:47,499 - radical.pilot.agent - INFO - RADICAL-Pilot multi-core agent for package/API version 0.18 2014-08-25 19:04:47,690 - radical.pilot.agent - INFO - Configured to run on system with PBSPRO. 2014-08-25 19:04:47,690 - radical.pilot.agent - INFO - Found PBSPro $PBS_NODEFILE /var/spool/PBS/aux/509101.sdb. 2014-08-25 19:04:47,745 - radical.pilot.agent - INFO - Discovered task launch command: '/opt/cray/alps/5.1.1-2.0501.8507.1.1.ari/bin/aprun' and MPI launch command: '/opt/cray/alps/5.1.1-2.0501.8507.1.1.ari/bin/a prun'. 2014-08-25 19:04:47,746 - radical.pilot.agent - INFO - Discovered execution environment: ['archer_2237', 'archer_2238', 'archer_2239'] 2014-08-25 19:04:48,133 - radical.pilot.agent - INFO - Started up <ExecWorker(ExecWorker-1, started daemon)> serving nodes ['archer_2237', 'archer_2238', 'archer_2239'] 2014-08-25 19:04:48,134 - radical.pilot.agent - INFO - Agent started. Database updated. 2014-08-25 19:04:48,410 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb777220a6413ee4e157d1 2014-08-25 19:04:49,147 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 0 2014-08-25 19:04:49,217 - radical.pilot.agent - INFO - Launching task 53fb777220a6413ee4e157d1 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb777220a6413ee4e157d1/radical_p ilot_cu_launch_script-qsGK6u.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb777220a6413ee4e157d1 2014-08-25 19:04:51,407 - radical.pilot.agent - INFO - Task 53fb777220a6413ee4e157d1 terminated with return code 1. 2014-08-25 19:04:51,928 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb777220a6413ee4e157d1/STDOUT to MongoDB as 53fb7ac3e896b9038120aec c. 2014-08-25 19:04:52,275 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb777220a6413ee4e157d1/STDERR to MongoDB as 53fb7ac3e896b9038120aec d. 2014-08-25 19:04:54,305 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ac520a6413ee4e157d2 2014-08-25 19:04:54,367 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 0 2014-08-25 19:04:54,394 - radical.pilot.agent - INFO - Launching task 53fb7ac520a6413ee4e157d2 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d2/radical_p ilot_cu_launch_script-Lqo6Xr.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d2 2014-08-25 19:04:55,484 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ac520a6413ee4e157d3 2014-08-25 19:04:55,489 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 1 2014-08-25 19:04:55,576 - radical.pilot.agent - INFO - Launching task 53fb7ac520a6413ee4e157d3 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d3/radical_p ilot_cu_launch_script-YFycas.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d3 2014-08-25 19:04:55,670 - radical.pilot.agent - INFO - Task 53fb7ac520a6413ee4e157d2 terminated with return code 1. 2014-08-25 19:04:55,931 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d2/STDOUT to MongoDB as 53fb7ac7e896b9038120aec f. 2014-08-25 19:04:56,278 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d2/STDERR to MongoDB as 53fb7ac7e896b9038120aed 0. 2014-08-25 19:04:56,663 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ac520a6413ee4e157d4 2014-08-25 19:04:57,369 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 0 2014-08-25 19:04:57,419 - radical.pilot.agent - INFO - Launching task 53fb7ac520a6413ee4e157d4 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d4/radical_p ilot_cu_launch_script-CyTsLe.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d4 2014-08-25 19:04:57,514 - radical.pilot.agent - INFO - Task 53fb7ac520a6413ee4e157d3 terminated with return code 1. 2014-08-25 19:04:57,775 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d3/STDOUT to MongoDB as 53fb7ac9e896b9038120aed 2. 2014-08-25 19:04:57,842 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ac520a6413ee4e157d5 2014-08-25 19:04:58,139 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d3/STDERR to MongoDB as 53fb7ac9e896b9038120aed 3. 2014-08-25 19:04:58,229 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 1 2014-08-25 19:04:58,284 - radical.pilot.agent - INFO - Launching task 53fb7ac520a6413ee4e157d5 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d5/radical_p ilot_cu_launch_script-_BzfJ0.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d5 2014-08-25 19:04:59,022 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ac520a6413ee4e157d6 2014-08-25 19:04:59,378 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 2 2014-08-25 19:04:59,436 - radical.pilot.agent - INFO - Launching task 53fb7ac520a6413ee4e157d6 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d6/radical_p ilot_cu_launch_script-PPgH0z.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d6 2014-08-25 19:04:59,530 - radical.pilot.agent - INFO - Task 53fb7ac520a6413ee4e157d4 terminated with return code 1. 2014-08-25 19:04:59,792 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d4/STDOUT to MongoDB as 53fb7acbe896b9038120aed 5. 2014-08-25 19:05:00,139 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d4/STDERR to MongoDB as 53fb7acbe896b9038120aed 6. 2014-08-25 19:05:00,141 - radical.pilot.agent - INFO - Task 53fb7ac520a6413ee4e157d5 terminated with return code 1. 2014-08-25 19:05:00,201 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ac520a6413ee4e157d7 2014-08-25 19:05:00,412 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d5/STDOUT to MongoDB as 53fb7acce896b9038120aed 8. 2014-08-25 19:05:00,759 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d5/STDERR to MongoDB as 53fb7acce896b9038120aed 9. 2014-08-25 19:05:00,760 - radical.pilot.agent - INFO - Task 53fb7ac520a6413ee4e157d6 terminated with return code 1. 2014-08-25 19:05:01,021 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d6/STDOUT to MongoDB as 53fb7acce896b9038120aed b. 2014-08-25 19:05:01,368 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d6/STDERR to MongoDB as 53fb7acde896b9038120aed c. 2014-08-25 19:05:01,381 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ac520a6413ee4e157d8 2014-08-25 19:05:01,636 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 0 2014-08-25 19:05:01,702 - radical.pilot.agent - INFO - Launching task 53fb7ac520a6413ee4e157d7 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d7/radical_p ilot_cu_launch_script-xJRn_n.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d7 2014-08-25 19:05:01,796 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 1 2014-08-25 19:05:01,829 - radical.pilot.agent - INFO - Launching task 53fb7ac520a6413ee4e157d8 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d8/radical_p ilot_cu_launch_script-OsFh2C.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d8 2014-08-25 19:05:02,560 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ac520a6413ee4e157d9 2014-08-25 19:05:02,923 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 2 2014-08-25 19:05:02,980 - radical.pilot.agent - INFO - Launching task 53fb7ac520a6413ee4e157d9 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d9/radical_p ilot_cu_launch_script-3BV5ts.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d9 2014-08-25 19:05:03,074 - radical.pilot.agent - INFO - Task 53fb7ac520a6413ee4e157d7 terminated with return code 1. 2014-08-25 19:05:03,335 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d7/STDOUT to MongoDB as 53fb7acfe896b9038120aed e. 2014-08-25 19:05:03,683 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d7/STDERR to MongoDB as 53fb7acfe896b9038120aed f. 2014-08-25 19:05:03,684 - radical.pilot.agent - INFO - Task 53fb7ac520a6413ee4e157d8 terminated with return code 1. 2014-08-25 19:05:03,948 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d8/STDOUT to MongoDB as 53fb7acfe896b9038120aee 1. 2014-08-25 19:05:04,295 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d8/STDERR to MongoDB as 53fb7acfe896b9038120aee 2. 2014-08-25 19:05:04,297 - radical.pilot.agent - INFO - Task 53fb7ac520a6413ee4e157d9 terminated with return code 1. 2014-08-25 19:05:04,558 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d9/STDOUT to MongoDB as 53fb7ad0e896b9038120aee 4. 2014-08-25 19:05:04,905 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ac520a6413ee4e157d9/STDERR to MongoDB as 53fb7ad0e896b9038120aee 5. 2014-08-25 19:05:14,349 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ad220a6413ee4e157da 2014-08-25 19:05:15,184 - radical.pilot.agent - INFO - Node archer_2237 satisfies 16 cores at offset 0 2014-08-25 19:05:15,253 - radical.pilot.agent - INFO - Launching task 53fb7ad220a6413ee4e157da via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ad220a6413ee4e157da/radical_p ilot_cu_launch_script-_TBpOQ.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ad220a6413ee4e157da 2014-08-25 19:05:17,360 - radical.pilot.agent - INFO - Task 53fb7ad220a6413ee4e157da terminated with return code 1. 2014-08-25 19:05:17,654 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ad220a6413ee4e157da/STDOUT to MongoDB as 53fb7adde896b9038120aee 7. 2014-08-25 19:05:18,049 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ad220a6413ee4e157da/STDERR to MongoDB as 53fb7adde896b9038120aee 8. 2014-08-25 19:05:19,156 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ade20a6413ee4e157e2 2014-08-25 19:05:20,152 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 0 2014-08-25 19:05:20,233 - radical.pilot.agent - INFO - Launching task 53fb7ade20a6413ee4e157e2 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e2/radical_p ilot_cu_launch_script-LGZubS.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e2 2014-08-25 19:05:20,383 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ade20a6413ee4e157db 2014-08-25 19:05:21,341 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 1 2014-08-25 19:05:21,401 - radical.pilot.agent - INFO - Launching task 53fb7ade20a6413ee4e157db via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157db/radical_p ilot_cu_launch_script-8fTQ7k.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157db 2014-08-25 19:05:21,508 - radical.pilot.agent - INFO - Task 53fb7ade20a6413ee4e157e2 terminated with return code 1. 2014-08-25 19:05:21,592 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ade20a6413ee4e157dc 2014-08-25 19:05:21,832 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e2/STDOUT to MongoDB as 53fb7ae1e896b9038120aee a. 2014-08-25 19:05:22,275 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e2/STDERR to MongoDB as 53fb7ae1e896b9038120aee b. 2014-08-25 19:05:22,378 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 0 2014-08-25 19:05:22,427 - radical.pilot.agent - INFO - Launching task 53fb7ade20a6413ee4e157dc via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157dc/radical_p ilot_cu_launch_script-cBMKUP.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157dc 2014-08-25 19:05:22,771 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ade20a6413ee4e157dd 2014-08-25 19:05:23,527 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 2 2014-08-25 19:05:23,586 - radical.pilot.agent - INFO - Launching task 53fb7ade20a6413ee4e157dd via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157dd/radical_p ilot_cu_launch_script-79ZgrO.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157dd 2014-08-25 19:05:23,680 - radical.pilot.agent - INFO - Task 53fb7ade20a6413ee4e157db terminated with return code 1. 2014-08-25 19:05:23,941 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157db/STDOUT to MongoDB as 53fb7ae3e896b9038120aee d. 2014-08-25 19:05:23,949 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ade20a6413ee4e157de 2014-08-25 19:05:24,290 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157db/STDERR to MongoDB as 53fb7ae3e896b9038120aee e. 2014-08-25 19:05:24,291 - radical.pilot.agent - INFO - Task 53fb7ade20a6413ee4e157dc terminated with return code 1. 2014-08-25 19:05:24,551 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157dc/STDOUT to MongoDB as 53fb7ae4e896b9038120aef 0. 2014-08-25 19:05:24,899 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157dc/STDERR to MongoDB as 53fb7ae4e896b9038120aef 1. 2014-08-25 19:05:24,900 - radical.pilot.agent - INFO - Task 53fb7ade20a6413ee4e157dd terminated with return code 1. 2014-08-25 19:05:25,128 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ade20a6413ee4e157df 2014-08-25 19:05:25,167 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157dd/STDOUT to MongoDB as 53fb7ae4e896b9038120aef 3. 2014-08-25 19:05:25,515 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157dd/STDERR to MongoDB as 53fb7ae5e896b9038120aef 4. 2014-08-25 19:05:25,783 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 0 2014-08-25 19:05:25,853 - radical.pilot.agent - INFO - Launching task 53fb7ade20a6413ee4e157de via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157de/radical_p ilot_cu_launch_script-boxdqH.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157de 2014-08-25 19:05:25,946 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 1 2014-08-25 19:05:25,984 - radical.pilot.agent - INFO - Launching task 53fb7ade20a6413ee4e157df via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157df/radical_p ilot_cu_launch_script-Lnybjw.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157df 2014-08-25 19:05:26,308 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ade20a6413ee4e157e0 2014-08-25 19:05:27,078 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 2 2014-08-25 19:05:27,127 - radical.pilot.agent - INFO - Launching task 53fb7ade20a6413ee4e157e0 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e0/radical_p ilot_cu_launch_script-pLryOd.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e0 2014-08-25 19:05:27,221 - radical.pilot.agent - INFO - Task 53fb7ade20a6413ee4e157de terminated with return code 1. 2014-08-25 19:05:27,482 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157de/STDOUT to MongoDB as 53fb7ae7e896b9038120aef 6. 2014-08-25 19:05:27,487 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7ade20a6413ee4e157e1 2014-08-25 19:05:27,830 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157de/STDERR to MongoDB as 53fb7ae7e896b9038120aef 7. 2014-08-25 19:05:27,831 - radical.pilot.agent - INFO - Task 53fb7ade20a6413ee4e157df terminated with return code 1. 2014-08-25 19:05:28,092 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157df/STDOUT to MongoDB as 53fb7ae7e896b9038120aef 9. 2014-08-25 19:05:28,439 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157df/STDERR to MongoDB as 53fb7ae8e896b9038120aef a. 2014-08-25 19:05:28,440 - radical.pilot.agent - INFO - Task 53fb7ade20a6413ee4e157e0 terminated with return code 1. 2014-08-25 19:05:28,703 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e0/STDOUT to MongoDB as 53fb7ae8e896b9038120aef c. 2014-08-25 19:05:29,050 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e0/STDERR to MongoDB as 53fb7ae8e896b9038120aef d. 2014-08-25 19:05:29,318 - radical.pilot.agent - INFO - Node archer_2237 satisfies 1 cores at offset 0 2014-08-25 19:05:29,365 - radical.pilot.agent - INFO - Launching task 53fb7ade20a6413ee4e157e1 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e1/radical_p ilot_cu_launch_script-gv7Owq.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e1 2014-08-25 19:05:31,460 - radical.pilot.agent - INFO - Task 53fb7ade20a6413ee4e157e1 terminated with return code 1. 2014-08-25 19:05:31,722 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e1/STDOUT to MongoDB as 53fb7aebe896b9038120aef f. 2014-08-25 19:05:32,069 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7ade20a6413ee4e157e1/STDERR to MongoDB as 53fb7aebe896b9038120af0 0. 2014-08-25 19:05:40,453 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53fb7aed20a6413ee4e157e3 2014-08-25 19:05:41,169 - radical.pilot.agent - INFO - Node archer_2237 satisfies 16 cores at offset 0 2014-08-25 19:05:41,236 - radical.pilot.agent - INFO - Launching task 53fb7aed20a6413ee4e157e3 via /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7aed20a6413ee4e157e3/radical_p ilot_cu_launch_script-HLL5eD.sh in /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7aed20a6413ee4e157e3 2014-08-25 19:05:43,332 - radical.pilot.agent - INFO - Task 53fb7aed20a6413ee4e157e3 terminated with return code 1. 2014-08-25 19:05:43,594 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7aed20a6413ee4e157e3/STDOUT to MongoDB as 53fb7af7e896b9038120af0 2. 2014-08-25 19:05:43,942 - radical.pilot.agent - INFO - Uploaded /fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf/unit-53fb7aed20a6413ee4e157e3/STDERR to MongoDB as 53fb7af7e896b9038120af0 3. 2014-08-25 19:05:45,088 - radical.pilot.agent - INFO - Received Cancel Pilot command. 2014-08-25 19:05:45,088 - radical.pilot.agent - WARNING - CANCEL received. Terminating.

AGENT.STDERR

ardi@eslogin001:/work/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf> more AGENT.STDERR

This is a private computing facility. Access to this system is limited to those who have been granted access by the operating service provider on behalf of the issuing authority and use is restricted to the purposes for which access was granted. All access and usage are governed by the terms and conditions of access agreed to by all registered users and are thus subject to the provisions of the Computer Misuse Act, 1990 under which unauthorised use is a criminal offence.

If you are not authorised to use this service you must disconnect immediately.

ardi@eslogin001:/work/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf>

AGENT.STDOUT

ardi@eslogin001:/work/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf> more AGENT.STDOUT

* ardi Job: 509101.sdb started: 25/08/14 19:04:46 host: mom3 * * ardi Job: 509101.sdb started: 25/08/14 19:04:46 host: mom3 * * ardi Job: 509101.sdb started: 25/08/14 19:04:46 host: mom3 * * ardi Job: 509101.sdb started: 25/08/14 19:04:46 host: mom3 *

################################################################################

Bootstrapper running on host: nid01921.

################################################################################

Environment of bootstrapper process:

LESSKEY=/etc/lesskey.bin MODULE_VERSION_STACK=3.2.6.7 CRAY_BINUTILS_BIN=/opt/cray/cce/8.2.6/cray-binutils/x86_64-unknown-linux-gnu/bin PE_LIBSCI_GENCOMPS_CRAY_sandybridge=81 PE_LIBSCI_VOLATILE_PRGENV=CRAY GNU INTEL MANPATH=/opt/cray/mpt/6.3.1/gni/man/mpich2:/opt/pbs/12.1.400.132424/man:/opt/cray/atp/1.7.2/man:/opt/cray/libsci/12.2.0/man:/opt/cray/cce/8.2.6/man:/opt/cray/cce/8.2.6/craylibs/man:/opt/cray/cce/8.2.6/CC/man:/op t/cray/cce/8.2.6/cftn/man:/opt/cray/craype/2.1.1/man:/opt/cray/llm/default/man:/opt/cray/lustre-cray_ari_s/2.4_3.0.80_0.5.1_1.0501.7664.13.1-1.0501.15783.26.1/man:/opt/cray/alps/5.1.1-2.0501.8507.1.1.ari/share/m an:/opt/modules/3.2.6.7/man:/usr/local/man:/usr/share/man:/usr/man:/opt/cray/share/man:/opt/intel/mic/share/man INFODIR=/usr/local/info:/usr/share/info:/usr/info NNTPSERVER=news PE_CXX_PKGCONFIG_LIBS=mpichcxx HOSTNAME=mom3 CRAY_UDREG_INCLUDE_OPTS=-I/opt/cray/udreg/2.3.2-1.0501.7914.1.13.ari/include GCC_X86_64=/opt/gcc/4.4.4/snos PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_sandybridge=130 PE_TRILINOS_DEFAULT_GENCOMPS_CRAY_x86_64=82 XKEYSYMDB=/usr/share/X11/XKeysymDB CRAY_SITE_LIST_DIR=/etc/opt/cray/modules LIBRARYMODULES=acml:alps:apprentice2:atp:cray-fftw:cray-libsci:cray-mpich2:cray-petsc:cray-petsc-complex:cray-shmem:cray-tpsl:cray-trilinos:cudatoolkit:fftw:ga:hdf5:hdf5-parallel:iobuf:lgdb:libfast:libsci_acc:mp ich1:mpich2:mrnet:netcdf:netcdf-hdf5parallel:netcdf-nofsync:netcdf-nofsync-hdf5parallel:ntk:onesided:papi:parallel-netcdf:petsc:petsc-complex:pmi:shmem:tpsl:trilinos:xt-atp:xt-lgdb:xt-libsci:xt-mpt:xt-papi:/etc/ opt/cray/modules/site_librarymodules RCLOCAL_BASEOPTS=true PE_NETCDF_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/netcdf/4.3.1/@PRGENV@/@PE_NETCDF_DEFAULT_GENCOMPS@/lib/pkgconfig PE_PARALLEL_NETCDF_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/parallel-netcdf/1.4.0/@PRGENV@/@PE_PARALLEL_NETCDF_DEFAULT_GENCOMPS@/lib/pkgconfig PE_TRILINOS_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/trilinos/11.6.1.0/@PRGENV@/@PE_TRILINOS_DEFAULT_GENCOMPS@/@PE_TRILINOS_DEFAULT_TARGET@/lib/pkgconfig PBS_ACCOUNT=e290 CRAY_BINUTILS_ROOT=/opt/cray/cce/8.2.6/cray-binutils CRAY_FTN_VERSION=8.2.6 PE_ENV=CRAY PE_LIBSCI_DEFAULT_GENCOMPS_GNU_interlagos=48 47 PE_SMA_VOLATILE_PKGCONFIG_PATH=/opt/cray/mpt/6.3.1/gni/sma@PE_SMA_DIR_DEFAULT64@/lib64/pkgconfig SHELL=/bin/bash HOST=mom3 ASSEMBLER_X86_64=/opt/cray/cce/8.2.6/cray-binutils/x86_64-unknown-linux-gnu/bin/as PKGCONFIG_ENABLED=1 HISTSIZE=1000 PROFILEREAD=true PE_LIBSCI_GENCOMPS_GNU_interlagos=48 47 PE_PETSC_DEFAULT_GENCOMPS_CRAY_sandybridge=81 XTOS_VERSION=5.1.29 PBS_JOBNAME=SAGA-Python-PBS TMPDIR=/tmp/pbs.509101.sdb CRAY_UGNI_POST_LINK_OPTS=-L/opt/cray/ugni/5.0-1.0501.8253.10.22.ari/lib64 CRAY_XPMEM_POST_LINK_OPTS=-L/opt/cray/xpmem/0.1-2.0501.48424.3.3.ari/lib64 CRAYPE_DIR=/opt/cray/craype/2.1.1 FORTRAN_SYSTEM_MODULE_NAMES=ftn_lib_definitions PE_NETCDF_DEFAULT_VOLATILE_PRGENV=GNU PE_PARALLEL_NETCDF_DEFAULT_VOLATILE_PRGENV=GNU PE_TPSL_DEFAULT_GENCOMPS_GNU_sandybridge=48 47 PE_TPSL_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI PE_TRILINOS_DEFAULT_GENCOMPS_CRAY_interlagos=82 PE_TRILINOS_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL PE_FFTW_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/fftw/3.3.0.4/@PE_FFTW_DEFAULT_TARGET@/lib/pkgconfig PE_HDF5_DEFAULT_VOLATILE_PRGENV=GNU PE_HDF5_PARALLEL_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/hdf5-parallel/1.8.12/@PRGENV@/@PE_HDF5_PARALLEL_DEFAULT_GENCOMPS@/lib/pkgconfig PE_NETCDF_HDF5PARALLEL_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/netcdf-hdf5parallel/4.3.1/@PRGENV@/@PE_NETCDF_HDF5PARALLEL_DEFAULT_GENCOMPS@/lib/pkgconfig PE_PETSC_DEFAULT_GENCOMPS_CRAY_interlagos=81 CRAY_MPICH2_DIR=/opt/cray/mpt/6.3.1/gni/mpich2-cray/81 PE_GA_DEFAULT_VOLATILE_PRGENV=GNU PE_LIBSCI_DEFAULT_GENCOMPS_GNU_x86_64=48 47 PBS_ENVIRONMENT=PBS_BATCH MORE=-sl PE_PKGCONFIG_PRODUCTS=PE_MPICH:PE_LIBSCI PE_TPSL_DEFAULT_GENCOMPS_INTEL_x86_64=130 PE_TRILINOS_DEFAULT_GENCOMPS_INTEL_interlagos=130 PE_MPICH_GENCOMPS_GNU=48 47 QTDIR=/usr/lib/qt3 PE_PAPI_DEFAULT_ACCEL_LIBS_nvidia35=-lcupti -lcudart -lcuda PE_PETSC_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI:PE_TPSL PE_CRAY_DEFAULT_FIXED_PKGCONFIG_PATH=/opt/cray/hdf5/1.8.12/CRAY/81/lib/pkgconfig:/opt/cray/ga/5.1.0.4/CRAY/81/lib/pkgconfig:/opt/cray/netcdf/4.3.1/CRAY/81/lib/pkgconfig:/opt/cray/parallel-netcdf/1.4.0/CRAY/81/li b/pkgconfig:/opt/cray/hdf5-parallel/1.8.12/CRAY/81/lib/pkgconfig:/opt/cray/netcdf-hdf5parallel/4.3.1/CRAY/81/lib/pkgconfig PBS_O_WORKDIR=/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf PE_PETSC_DEFAULT_GENCOMPS_CRAY_x86_64=81 PE_FORTRAN_PKGCONFIG_LIBS=mpichf90 NCPUS=1 CRAY_BINUTILS_VERSION=/opt/cray/cce/8.2.6 CRAY_PRGENVCRAY=loaded PE_TRILINOS_DEFAULT_GENCOMPS_GNU_interlagos=48 47 NODE_COUNT=3 PE_LIBSCI_GENCOMPS_CRAY_interlagos=81 PE_TRILINOS_DEFAULT_GENCOMPS_INTEL_x86_64=130 USER=ardi PBS_TASKNUM=1 JRE_HOME=/usr/lib64/jvm/jre BUILD_OPTS=/opt/cray/craype/2.1.1/bin/build-opts PE_SMA_DIR_CRAY_DEFAULT64=64 LS_COLORS= PE_FFTW_DEFAULT_TARGET_interlagos=interlagos PE_LIBSCI_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL PE_TPSL_DEFAULT_GENCOMPS_CRAY_x86_64=81 PBS_O_HOME=/home/e290/e290/ardi CRAY_RCA_POST_LINK_OPTS=-L/opt/cray/rca/1.0.0-2.0501.48090.7.46.ari/lib64 -lrca PE_PETSC_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL PE_PKGCONFIG_LIBS=mpich:AtpSigHandler:sci_mpi_mp:sci_mp PE_MPICH_FIXED_PRGENV=INTEL XNLSPATH=/usr/share/X11/nls FTN_X86_64=/opt/cray/cce/8.2.6/cftn/x86-64 PE_PETSC_DEFAULT_GENCOMPS_GNU_interlagos=48 47 PE_PETSC_DEFAULT_GENCOMPS_GNU_sandybridge=48 47 PE_PETSC_DEFAULT_GENCOMPS_INTEL_interlagos=130 PE_PETSC_DEFAULT_GENCOMPS_INTEL_sandybridge=130 NUM_PES=72 ENV=/etc/bash.bashrc MPICH_ABORT_ON_ERROR=1 PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_x86_64=81 PE_PAPI_DEFAULT_PKGCONFIG_VARIABLES=PE_PAPI_ACCELLIBS@accelerator@ MPICH_DIR=/opt/cray/mpt/6.3.1/gni/mpich2-cray/81 HOSTTYPE=x86_64 ATP_POST_LINK_OPTS=-Wl,-L/opt/cray/atp/1.7.2/lib/ PE_FFTW_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH PE_FFTW_DEFAULT_TARGET_sandybridge=sandybridge RCLOCAL_PRGENV=true PBS_MOMPORT=15003 FROM_HEADER= PE_PRODUCT_LIST=CRAYPE_IVYBRIDGE:CRAY_RCA:CRAY_PMI:CRAY_LIBSCI:CRAYPE:CRAY:CRAY_LLM:CRAY_XPMEM:CRAY_DMAPP:CRAY_UGNI:CRAY_UDREG:CRAY_ALPS PE_LIBSCI_GENCOMPS_INTEL_x86_64=130 PE_TPSL_DEFAULT_GENCOMPS_GNU_interlagos=48 47 PE_TRILINOS_DEFAULT_GENCOMPS_CRAY_sandybridge=82 PAGER=less FFTW_SYSTEM_WISDOM_DIR=/opt/cray/libsci/12.2.0 PE_LIBSCI_GENCOMPS_GNU_sandybridge=48 47 CSHEDIT=emacs NUM_PPN=24 PBS_O_QUEUE=standard XDG_CONFIG_DIRS=/etc/xdg PE_LIBSCI_GENCOMPS_CRAY_x86_64=81 PE_MPICH_DEFAULT_VOLATILE_PRGENV=CRAY GNU PE_MPICH_TARGET_VAR_nvidia20=-lcudart PE_TPSL_DEFAULT_GENCOMPS_CRAY_sandybridge=81 MINICOM=-c on USERMODULES=acml:alps:apprentice2:atp:blcr:cce:chapel:cray-fftw:cray-libsci:cray-mpich2:craypat:craype:cray-petsc:cray-petsc-complex:cray-shmem:cray-tpsl:cray-trilinos:cudatoolkit:ddt:fftw:ga:gcc:hdf5:hdf5-paral lel:intel:iobuf:java:lgdb:libfast:libsci_acc:mpich1:mrnet:netcdf:netcdf-hdf5parallel:netcdf-nofsync:netcdf-nofsync-hdf5parallel:ntk:onesided:papi:parallel-netcdf:pathscale:perftools:petsc:petsc-complex:pgi:pmi:P rgEnv-cray:PrgEnv-gnu:PrgEnv-intel:PrgEnv-pathscale:PrgEnv-pgi:stat:totalview:tpsl:trilinos:xt-asyncpe:xt-craypat:xt-lgdb:xt-libsci:xt-mpich2:xt-mpt:xt-papi:xt-shmem:xt-totalview:/etc/opt/cray/modules/site_userm odules CRAY_DMAPP_INCLUDE_OPTS=-I/opt/cray/dmapp/7.0.1-1.0501.8315.8.4.ari/include -I/opt/cray/gni-headers/3.0-1.0501.8317.12.1.ari/include CRAY_LIBSCI_BASE_DIR=/opt/cray/libsci/12.2.0 CRAY_LIBSCI_DIR=/opt/cray/libsci/12.2.0 NLSPATH=/opt/cray/cce/8.2.6/CC/x86-64/nls/En/%N.cat:/opt/cray/cce/8.2.6/craylibs/x86-64/nls/En/%N.cat:/opt/cray/cce/8.2.6/cftn/x86-64/nls/En/%N.cat PE_LIBSCI_PKGCONFIG_LIBS=sci_mpi_mp:sci_mp PE_NETCDF_DEFAULT_GENCOMPS_GNU=48 47 PE_PARALLEL_NETCDF_DEFAULT_GENCOMPS_GNU=48 47 PATH=/opt/pbs/12.1.400.132424/bin:/opt/cray/atp/1.7.2/bin:/opt/cray/rca/1.0.0-2.0501.48090.7.46.ari/bin:/opt/cray/pmi/5.0.3-1.0000.9981.128.2.ari/bin:/opt/cray/cce/8.2.6/cray-binutils/x86_64-unknown-linux-gnu/bi n:/opt/cray/cce/8.2.6/craylibs/x86-64/bin:/opt/cray/cce/8.2.6/cftn/bin:/opt/cray/cce/8.2.6/CC/bin:/opt/cray/craype/2.1.1/bin:/opt/cray/llm/default/bin:/opt/cray/llm/default/etc:/opt/cray/xpmem/0.1-2.0501.48424.3 .3.ari/bin:/opt/cray/dmapp/7.0.1-1.0501.8315.8.4.ari/bin:/opt/cray/ugni/5.0-1.0501.8253.10.22.ari/bin:/opt/cray/udreg/2.3.2-1.0501.7914.1.13.ari/bin:/opt/cray/lustre-cray_ari_s/2.4_3.0.80_0.5.1_1.0501.7664.13.1- 1.0501.15783.26.1/sbin:/opt/cray/lustre-cray_ari_s/2.4_3.0.80_0.5.1_1.0501.7664.13.1-1.0501.15783.26.1/bin:/opt/cray/MySQL/5.0.64-1.0000.7096.23.2/sbin:/opt/cray/MySQL/5.0.64-1.0000.7096.23.2/bin:/opt/cray/alps/ 5.1.1-2.0501.8507.1.1.ari/sbin:/opt/cray/alps/5.1.1-2.0501.8507.1.1.ari/bin:/opt/cray/sdb/1.0-1.0501.48084.4.48.ari/bin:/opt/cray/nodestat/2.2-1.0501.47138.1.78.ari/bin:/opt/modules/3.2.6.7/bin:/usr/local/bin:/u sr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib64/jvm/jre/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/usr/lib/qt3/bin:/opt/cray/bin PBS_O_LOGNAME=ardi MAIL=/var/spool/mail/ardi MODULE_VERSION=3.2.6.7 PE_HDF5_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/hdf5/1.8.12/@PRGENV@/@PE_HDF5_DEFAULT_GENCOMPS@/lib/pkgconfig PE_PKGCONFIG_DEFAULT_PRODUCTS=PE_HDF5:PE_TPSL:PE_GA:PE_NETCDF:PE_PARALLEL_NETCDF:PE_TRILINOS:PE_HDF5_PARALLEL:PE_NETCDF_HDF5PARALLEL:PE_FFTW:PE_LIBSCI:PE_MPICH:PE_PETSC PE_TPSL_DEFAULT_GENCOMPS_CRAY_interlagos=81 PBS_O_LANG=en_US.UTF-8 CPU=x86_64 XTPE_NETWORK_TARGET=aries PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_sandybridge=81 PBS_JOBCOOKIE=000000005AFD83D1000000000F0ABC9C JAVA_BINDIR=/usr/lib64/jvm/jre/bin CRAY_PE_TARGET=x86-64 PE_HDF5_PARALLEL_DEFAULT_FIXED_PRGENV=CRAY INTEL PE_HDF5_PARALLEL_DEFAULT_GENCOMPS_GNU=48 47 PE_NETCDF_HDF5PARALLEL_DEFAULT_FIXED_PRGENV=CRAY INTEL PE_NETCDF_HDF5PARALLEL_DEFAULT_GENCOMPS_GNU=48 47 CRAY_UDREG_POST_LINK_OPTS=-L/opt/cray/udreg/2.3.2-1.0501.7914.1.13.ari/lib64 PE_TPSL_DEFAULT_GENCOMPS_INTEL_interlagos=130 PWD=/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-53fb776f20a6413ee4e157cf INPUTRC=/etc/inputrc CRAY_ALPS_POST_LINK_OPTS=-L/opt/cray/alps/5.1.1-2.0501.8507.1.1.ari/lib64 CRAYPE_VERSION=2.1.1 PE_TRILINOS_DEFAULT_GENCOMPS_GNU_sandybridge=48 47 PE_MPICH_VOLATILE_PRGENV=CRAY GNU JAVA_HOME=/usr/lib64/jvm/jre TARGETMODULES=craype-abudhabi:craype-abudhabi-cu:craype-accel-nvidia20:craype-accel-nvidia30:craype-accel-nvidia35:craype-barcelona:craype-hugepages128K:craype-hugepages128M:craype-hugepages16M:craype-hugepages2 56M:craype-hugepages2M:craype-hugepages512K:craype-hugepages512M:craype-hugepages64M:craype-hugepages8M:craype-interlagos:craype-interlagos-cu:craype-istanbul:craype-ivybridge:craype-knc:craype-mc12:craype-mc8:c raype-network-aries:craype-network-gemini:craype-network-seastar:craype-sandybridge:craype-shanghai:craype-target-compute_node:craype-target-local_host:craype-target-native:craype-target-petest:craype-xeon:xtpe- barcelona:xtpe-interlagos:xtpe-interlagos-cu:xtpe-istanbul:xtpe-mc12:xtpe-mc8:xtpe-network-gemini:xtpe-network-seastar:xtpe-shanghai:xtpe-target-native:xtpe-xeon:/etc/opt/cray/modules/site_targetmodules LMFILES=/opt/modulefiles/modules/3.2.6.7:/opt/cray/ari/modulefiles/nodestat/2.2-1.0501.47138.1.78.ari:/opt/cray/ari/modulefiles/sdb/1.0-1.0501.48084.4.48.ari:/opt/cray/ari/modulefiles/alps/5.1.1-2.0501.8507.1. 1.ari:/opt/cray/modulefiles/MySQL/5.0.64-1.0000.7096.23.2:/opt/cray/modulefiles/lustre-cray_ari_s/2.4_3.0.80_0.5.1_1.0501.7664.13.1-1.0501.15783.26.1:/opt/cray/ari/modulefiles/udreg/2.3.2-1.0501.7914.1.13.ari:/o pt/cray/ari/modulefiles/ugni/5.0-1.0501.8253.10.22.ari:/opt/cray/ari/modulefiles/gni-headers/3.0-1.0501.8317.12.1.ari:/opt/cray/ari/modulefiles/dmapp/7.0.1-1.0501.8315.8.4.ari:/opt/cray/ari/modulefiles/xpmem/0.1 -2.0501.48424.3.3.ari:/opt/modulefiles/hss-llm/7.1.0:/opt/modulefiles/Base-opts/1.0.2-1.0501.47945.4.2.ari:/opt/cray/craype/default/modulefiles/craype-network-aries:/opt/cray/modulefiles/craype/2.1.1:/opt/module files/cce/8.2.6:/opt/cray/modulefiles/cray-libsci/12.2.0:/opt/cray/ari/modulefiles/pmi/5.0.3-1.0000.9981.128.2.ari:/opt/cray/ari/modulefiles/rca/1.0.0-2.0501.48090.7.46.ari:/opt/cray/modulefiles/atp/1.7.2:/opt/c ray/modulefiles/PrgEnv-cray/5.1.29:/opt/modulefiles/pbs/12.1.400.132424:/opt/cray/craype/default/modulefiles/craype-ivybridge:/opt/cray/modulefiles/cray-mpich/6.3.1:/opt/modulefiles/packages-archer INCLUDE_PATH_X86_64=/opt/cray/cce/8.2.6/craylibs/x86-64/include PE_MPICH_DEFAULT_GENCOMPS_CRAY=81 PE_LIBSCI_DEFAULT_GENCOMPS_GNU_sandybridge=48 47 PE_LIBSCI_GENCOMPS_INTEL_interlagos=130 PBS_NODENUM=0 LANG=en_US.UTF-8 PE_INTEL_FIXED_PKGCONFIG_PATH=/opt/cray/mpt/6.3.1/gni/mpich2-intel/130/lib/pkgconfig PYTHONSTARTUP=/etc/pythonstart MODULEPATH=/opt/cray/craype/default/modulefiles:/opt/cray/ari/modulefiles:/opt/cray/modulefiles:/opt/modulefiles:/opt/modules/packages-archer PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_interlagos=130 PE_MPICH_NV_LIBS_nvidia20=-lcudart PE_MPICH_VOLATILE_PKGCONFIG_PATH=/opt/cray/mpt/6.3.1/gni/mpich2-@PRGENV@@PE_MPICH_DIR_DEFAULT64@/@PE_MPICH_GENCOMPS@/lib/pkgconfig TZ=Europe/London NUM_DEPTH=1 PBS_JOBDIR=/home/e290/e290/ardi LOADEDMODULES=modules/3.2.6.7:nodestat/2.2-1.0501.47138.1.78.ari:sdb/1.0-1.0501.48084.4.48.ari:alps/5.1.1-2.0501.8507.1.1.ari:MySQL/5.0.64-1.0000.7096.23.2:lustre-cray_ari_s/2.4_3.0.80_0.5.1_1.0501.7664.13.1-1.0 501.15783.26.1:udreg/2.3.2-1.0501.7914.1.13.ari:ugni/5.0-1.0501.8253.10.22.ari:gni-headers/3.0-1.0501.8317.12.1.ari:dmapp/7.0.1-1.0501.8315.8.4.ari:xpmem/0.1-2.0501.48424.3.3.ari:hss-llm/7.1.0:Base-opts/1.0.2-1. 0501.47945.4.2.ari:craype-network-aries:craype/2.1.1:cce/8.2.6:cray-libsci/12.2.0:pmi/5.0.3-1.0000.9981.128.2.ari:rca/1.0.0-2.0501.48090.7.46.ari:atp/1.7.2:PrgEnv-cray/5.1.29:pbs/12.1.400.132424:craype-ivybridge :cray-mpich/6.3.1:packages-archer SHMEM_ABORT_ON_ERROR=1 CRAY_DMAPP_POST_LINK_OPTS=-L/opt/cray/dmapp/7.0.1-1.0501.8315.8.4.ari/lib64 PBS_O_SHELL=/bin/bash CRAY_RCA_INCLUDE_OPTS=-I/opt/cray/rca/1.0.0-2.0501.48090.7.46.ari/include -I/opt/cray-hss-devel/7.1.0/include -I/opt/cray/krca/1.0.0-2.0501.47640.3.70.ari/include PE_SMA_DIR_PGI_DEFAULT64=64 PBS_JOBID=509101.sdb PE_LIBSCI_DEFAULT_GENCOMPS_INTEL_x86_64=130 PE_MPICH_PKGCONFIG_VARIABLES=PE_MPICH_NVLIBS@accelerator@ ENVIRONMENT=BATCH CRAY_CC_VERSION=8.2.6 CRAY_PMI_POST_LINK_OPTS=-L/opt/cray/pmi/5.0.3-1.0000.9981.128.2.ari/lib64 PE_HDF5_DEFAULT_FIXED_PRGENV=CRAY INTEL PE_TPSL_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/tpsl/1.4.0/@PRGENV@/@PE_TPSL_DEFAULT_GENCOMPS@/@PE_TPSL_DEFAULT_TARGET@/lib/pkgconfig CRAY_MPICH2_VER=6.3.1 PE_LIBSCI_VOLATILE_PKGCONFIG_PATH=/opt/cray/libsci/12.2.0/@PRGENV@/@PE_LIBSCI_GENCOMPS@/@PE_LIBSCI_TARGET@/lib/pkgconfig PE_NETCDF_DEFAULT_FIXED_PRGENV=CRAY INTEL PE_PARALLEL_NETCDF_DEFAULT_FIXED_PRGENV=CRAY INTEL HOME=/home/e290/e290/ardi SHLVL=4 QT_SYSTEM_DIR=/usr/share/desktop-data CRAY_LIBSCI_VERSION=12.2.0 PE_HDF5_PARALLEL_DEFAULT_VOLATILE_PRGENV=GNU PE_MPICH_TARGET_VAR_nvidia35=-lcudart PE_NETCDF_HDF5PARALLEL_DEFAULT_VOLATILE_PRGENV=GNU PE_PKGCONFIG_PRODUCTS_DEFAULT=PE_PAPI OSTYPE=linux LESS_ADVANCED_PREPROCESSOR=no LINKER_X86_64=/opt/cray/cce/8.2.6/cray-binutils/x86_64-unknown-linux-gnu/bin/ld PE_MPICH_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/mpt/6.3.1/gni/mpich2-@PRGENV@@PE_MPICH_DEFAULT_DIR_DEFAULT64@/@PE_MPICH_DEFAULT_GENCOMPS@/lib/pkgconfig PE_TPSL_DEFAULT_VOLATILE_PRGENV=CRAY GNU INTEL PBS_O_HOST=eslogin1-ldap XCURSOR_THEME=crystalwhite LS_OPTIONS=-N --color=none -T 0 CRAY_PMI_INCLUDE_OPTS=-I/opt/cray/pmi/5.0.3-1.0000.9981.128.2.ari/include PE_TPSL_DEFAULT_GENCOMPS_INTEL_sandybridge=130 CRAY_MPICH2_BASEDIR=/opt/cray/mpt/6.3.1/gni WINDOWMANAGER= PRGENVMODULES=PrgEnv-cray:PrgEnv-gnu:PrgEnv-intel:PrgEnv-pathscale:PrgEnv-pgi CRAY_LLM_DIR=/opt/cray/llm/default CRAYPE_NETWORK_TARGET=aries ATP_MRNET_COMM_PATH=/opt/cray/atp/1.7.2/bin/atp_mrnet_commnode_wrapper CRAYLMD_LICENSE_FILE=/opt/cray/cce/cce.lic PKG_CONFIG_PATH_DEFAULT=/opt/cray/papi/5.3.0/lib64/pkgconfig PE_MPICH_DIR_CRAY_DEFAULT64=64 PE_LEVEL=8.2 PE_PAPI_DEFAULT_TARGET_VAR_nvidia35=-lcupti -lcudart -lcuda LOGNAME=ardi MACHTYPE=x86_64-suse-linux LESS=-M -I G_FILENAME_ENCODING=@locale,UTF-8,ISO-8859-15,CP1252 CRAY_GNI_HEADERS_INCLUDE_OPTS=-I/opt/cray/gni-headers/3.0-1.0501.8317.12.1.ari/include PYTHONPATH=/opt/cray/sdb/1.0-1.0501.48084.4.48.ari/lib64/py CRAYLIBS_X86_64=/opt/cray/cce/8.2.6/craylibs/x86-64 CRAY_LIBSCI_PREFIX_DIR=/opt/cray/libsci/12.2.0/CRAY/81/x86_64 PE_HDF5_DEFAULT_GENCOMPS_GNU=48 47 PE_MPICH_NV_LIBS= PE_TPSL_DEFAULT_GENCOMPS_GNU_x86_64=48 47 PE_TRILINOS_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_HDF5_PARALLEL:PE_NETCDF_HDF5PARALLEL:PE_LIBSCI:PE_TPSL CVS_RSH=ssh DMAPP_ABORT_ON_ERROR=1 PE_TRILINOS_DEFAULT_GENCOMPS_GNU_x86_64=48 47 PE_MPICH_GENCOMPS_CRAY=81 PBS_QUEUE=standard XDG_DATA_DIRS=/usr/share:/etc/opt/kde3/share:/opt/kde3/share TOOLMODULES=apprentice:apprentice2:atp:chapel:craypat:ddt:gdb:iobuf:mrnet:papi:perftools:stat:totalview:xt-craypat:xt-lgdb:xt-papi:xt-totalview:/etc/opt/cray/modules/site_toolmodules PE_LIBSCI_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH PE_LIBSCI_GENCOMPS_INTEL_sandybridge=130 PE_MPICH_DEFAULT_FIXED_PRGENV=INTEL PE_MPICH_DEFAULT_GENCOMPS_GNU=48 47 MODULESHOME=/opt/modules/3.2.6.7 PE_GA_DEFAULT_FIXED_PRGENV=CRAY INTEL PE_LIBSCI_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/libsci/12.2.0/@PRGENV@/@PE_LIBSCI_DEFAULT_GENCOMPS@/@PE_LIBSCI_DEFAULT_TARGET@/lib/pkgconfig PBS_O_MAIL=/var/mail/ardi OMP_NUM_THREADS=1 LESSOPEN=lessopen.sh %s PKG_CONFIG_PATH=/opt/cray/rca/1.0.0-2.0501.48090.7.46.ari/lib64/pkgconfig:/opt/cray/pmi/5.0.3-1.0000.9981.128.2.ari/lib64/pkgconfig:/opt/cray/iobuf/2.0.5/lib/pkgconfig:/opt/cray/xpmem/0.1-2.0501.48424.3.3.ari/li b64/pkgconfig:/opt/cray/dmapp/7.0.1-1.0501.8315.8.4.ari/lib64/pkgconfig:/opt/cray/gni-headers/3.0-1.0501.8317.12.1.ari/lib64/pkgconfig:/opt/cray/ugni/5.0-1.0501.8253.10.22.ari/lib64/pkgconfig:/opt/cray/udreg/2.3 .2-1.0501.7914.1.13.ari/lib64/pkgconfig:/opt/cray/MySQL/5.0.64-1.0000.7096.23.2/lib64/pkgconfig:/opt/cray/alps/5.1.1-2.0501.8507.1.1.ari/lib64/pkgconfig:/opt/cray/atp/1.7.2/lib/pkgconfig PE_MPICH_NV_LIBS_nvidia35=-lcudart PE_PETSC_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/petsc/3.4.3.1/real/@PRGENV@/@PE_PETSC_DEFAULT_GENCOMPS@/@PE_PETSC_DEFAULT_TARGET@/lib/pkgconfig PELOCAL_PRGENV=true LIBSCI_BASE_DIR=/opt/cray/libsci/12.2.0 INFOPATH=/usr/local/info:/usr/share/info:/usr/info LIBSCI_VERSION=12.2.0 CRAY_MPICH2_ROOTDIR=/opt/cray/mpt/6.3.1 CRAY_ALPS_INCLUDE_OPTS=-I/opt/cray/alps/5.1.1-2.0501.8507.1.1.ari/include CRAY_PRE_COMPILE_OPTS=-hnetwork=aries PE_LIBSCI_DEFAULT_GENCOMPS_CRAY_interlagos=81 CRAY_CPU_TARGET=ivybridge CRAY_UGNI_INCLUDE_OPTS=-I/opt/cray/ugni/5.0-1.0501.8253.10.22.ari/include CRAY_XPMEM_INCLUDE_OPTS=-I/opt/cray/xpmem/0.1-2.0501.48424.3.3.ari/include PE_LIBSCI_REQUIRED_PRODUCTS=PE_MPICH PE_LIBSCI_GENCOMPS_GNU_x86_64=48 47 PBS_O_SYSTEM=Linux LESSCLOSE=lessclose.sh %s %s ATP_HOME=/opt/cray/atp/1.7.2 PE_FFTW_DEFAULT_TARGET_x86_64=x86_64 PE_TRILINOS_DEFAULT_GENCOMPS_INTEL_sandybridge=130 PBS_NODEFILE=/var/spool/PBS/aux/509101.sdb G_BROKEN_FILENAMES=1 CRAY_LD_LIBRARY_PATH=/opt/cray/mpt/6.3.1/gni/mpich2-cray/81/lib:/opt/cray/rca/1.0.0-2.0501.48090.7.46.ari/lib64:/opt/cray/pmi/5.0.3-1.0000.9981.128.2.ari/lib64:/opt/cray/libsci/12.2.0/CRAY/81/x86_64/lib:/opt/cra y/cce/8.2.6/CC/x86-64/lib/x86-64:/opt/cray/cce/8.2.6/craylibs/x86-64:/opt/cray/xpmem/0.1-2.0501.48424.3.3.ari/lib64:/opt/cray/dmapp/7.0.1-1.0501.8315.8.4.ari/lib64:/opt/cray/ugni/5.0-1.0501.8253.10.22.ari/lib64: /opt/cray/udreg/2.3.2-1.0501.7914.1.13.ari/lib64:/opt/cray/alps/5.1.1-2.0501.8507.1.1.ari/lib64 CC_X86_64=/opt/cray/cce/8.2.6/CC/x86-64 CRAYOS_VERSION=5.1.29 PE_GA_DEFAULT_GENCOMPS_GNU=48 47 PE_GA_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/ga/5.1.0.4/@PRGENV@/@PE_GA_DEFAULT_GENCOMPS@/lib/pkgconfig PE_INTEL_DEFAULT_FIXED_PKGCONFIG_PATH=/opt/cray/hdf5/1.8.12/INTEL/130/lib/pkgconfig:/opt/cray/ga/5.1.0.4/INTEL/130/lib/pkgconfig:/opt/cray/netcdf/4.3.1/INTEL/130/lib/pkgconfig:/opt/cray/parallel-netcdf/1.4.0/INT EL/130/lib/pkgconfig:/opt/cray/hdf5-parallel/1.8.12/INTEL/130/lib/pkgconfig:/opt/cray/netcdf-hdf5parallel/4.3.1/INTEL/130/lib/pkgconfig:/opt/cray/mpt/6.3.1/gni/mpich2-intel/130/lib/pkgconfig PE_PAPI_DEFAULT_ACCEL_LIBS= PBS_O_PATH=/usr/local/packages/cse/imagemagick/6.8.8-2/bin:/home/y07/y07/cse/nano/2.2.6/bin:/home/y07/y07/cse/tkdiff/4.2:/work/y07/y07/cse/python/2.7.6/bin:/usr/local/packages/cse/serialJobs:/usr/local/packages/ cse/bolt/bin:/usr/local/packages/cse/checkScript:/usr/local/packages/cse/budgets:/opt/pbs/12.1.400.132424/bin:/opt/cray/atp/1.7.2/bin:/opt/cray/rca/1.0.0-2.0501.48090.7.46.ari/bin:/opt/cray/alps/5.1.1-2.0501.850 7.1.1.ari/sbin:/opt/cray/alps/5.1.1-2.0501.8507.1.1.ari/bin:/opt/cray/dvs/2.4_0.9.0-1.0501.1672.2.122.ari/bin:/opt/cray/csa/3.0.0-1_2.0501.47112.1.91.ari/sbin:/opt/cray/csa/3.0.0-1_2.0501.47112.1.91.ari/bin:/opt /cray/job/1.5.5-0.1_2.0501.48066.2.43.ari/bin:/opt/cray/xpmem/0.1-2.0501.48424.3.3.ari/bin:/opt/cray/dmapp/7.0.1-1.0501.8315.8.4.ari/bin:/opt/cray/pmi/5.0.3-1.0000.9981.128.2.ari/bin:/opt/cray/ugni/5.0-1.0501.82 53.10.22.ari/bin:/opt/cray/udreg/2.3.2-1.0501.7914.1.13.ari/bin:/opt/cray/cce/8.2.6/cray-binutils/x86_64-unknown-linux-gnu/bin:/opt/cray/cce/8.2.6/craylibs/x86-64/bin:/opt/cray/cce/8.2.6/cftn/bin:/opt/cray/cce/8 .2.6/CC/bin:/opt/cray/craype/2.1.1/bin:/opt/cray/switch/1.0-1.0501.47124.1.93.ari/bin:/opt/cray/eslogin/eswrap/1.1.0-1.010400.915.0/bin:/opt/modules/3.2.6.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11 R6/bin:/usr/games:/usr/lib64/jvm/jre/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/sbin:/usr/sbin:.:/usr/lib/qt3/bin:/opt/cray/bin COLORTERM=1 JAVA_ROOT=/usr/lib64/jvm/jre PE_MPICH_DEFAULT_DIR_CRAY_DEFAULT64=64 PE_PETSC_DEFAULT_GENCOMPS_GNU_x86_64=48 47 PE_PETSC_DEFAULT_GENCOMPS_INTEL_x8664=130 =/usr/bin/printenv

################################################################################

Running pre-bootstrapping command

CMDLINE: module load python

################################################################################

Setting up forward tunnel for MongoDB to 10.60.0.52.

################################################################################

Searching for available TCP port for tunnel in range 23000..23100.

Found available port: 23000

################################################################################

Launching radical-pilot-agent for 64 cores.

CMDLINE: python radical-pilot-agent.py -b 0 -c 64 -d 20 -j APRUN -k APRUN -l PBSPRO -m mongodb://127.0.0.1:23000 -n radicalpilot -p 53fb776f20a6413ee4e157cf -s 53fb776e20a6413ee4

e157cd -t 60 -v 0.18

Resources requested: ncpus=72,place=free,walltime=01:00:00 Resources allocated: cpupercent=0,cput=00:00:02,mem=38212kb,ncpus=72,vmem=510268kb,walltime=00:01:00

* ardi Job: 509101.sdb ended: 25/08/14 19:05:45 queue: standard * * ardi Job: 509101.sdb ended: 25/08/14 19:05:45 queue: standard * * ardi Job: 509101.sdb ended: 25/08/14 19:05:45 queue: standard *

* ardi Job: 509101.sdb ended: 25/08/14 19:05:45 queue: standard *

This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.

radical-cybertools / ExTASY