getting_started_local.py hangs

danielskatz commented 9 years ago

Here's what happens:

(sagenv)dis-dlien-1:examples dsk$ python getting_started_local.py [Callback]: ComputePilot '53dfeeb71d2f566368b50f02' state changed to Launching. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f04' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f05' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f06' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f07' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f08' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f09' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0a' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0b' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0c' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0d' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0e' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0f' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f10' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f11' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f12' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f13' state changed to PendingInputTransfer. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f04' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f05' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f06' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f07' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f05' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f04' state changed to PendingExecution. [Callback]: ComputePilot '53dfeeb71d2f566368b50f02' state changed to PendingActive. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f08' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f09' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f06' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f07' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0a' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0b' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f08' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f09' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0c' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0d' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0a' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0b' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0e' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0f' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0c' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0d' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f10' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f11' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0e' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f0f' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f12' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f13' state changed to TransferringInput. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f10' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f11' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f12' state changed to PendingExecution. [Callback]: ComputeUnit '53dfeeb71d2f566368b50f13' state changed to PendingExecution.

(after this, nothing more happens - I've waited about 10 minutes - then I kill the job)

Here's the traceback: Traceback (most recent call last): File "getting_started_local.py", line 119, in umgr.wait_units() Process InputFileTransferWorker-1: Process OutputFileTransferWorker-1: Process OutputFileTransferWorker-2: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap Traceback (most recent call last): Process InputFileTransferWorker-2: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap Process PilotLauncherWorker-1: Traceback (most recent call last): self.run() File "/Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/controller/input_file_transfer_worker.py", line 79, in run File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap time.sleep(1) self.run() KeyboardInterrupt File "/Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/controller/pilot_launcher_worker.py", line 153, in run time.sleep(1) KeyboardInterrupt self.run() File "/Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/controller/output_file_transfer_worker.py", line 79, in run self.run() File "/Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/controller/output_file_transfer_worker.py", line 79, in run time.sleep(1) KeyboardInterrupt File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/controller/input_file_transfer_worker.py", line 79, in run time.sleep(1) time.sleep(1) KeyboardInterrupt KeyboardInterrupt File "/Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/unit_manager.py", line 535, in wait_units time.sleep(1) KeyboardInterrupt

andre-merzky commented 9 years ago

Dan, thanks for your tickets in general, those are useful.

For this specific one: we are not well reporting pilot activity during bootstrap at the moment. For compute units, we have a mechanism in place to report stderr on failing units -- on pilots this is not yet working. The only way to find out what happens is, at this point, to look into ~/radical.pilot.sandbox/pilot-/ -- there are some log files and stdout/stderr of the pilot.

On a very slow network link, the bootstrapping of the pilot can take significant time, as it pulls a number of python packages from the pypi repository. I frequently see startup times longer than 2 minutes on my 6MB line (which has a high latency though). 10min though seem extraordinary long.

If you don't mind, please let us know what you find in the output and log files, and please let us know on what kind of setup you are working.

danielskatz commented 9 years ago

note that this is a local test. should the network matter?

stdout ends with

################################################################################

Launching radical-pilot-agent for 2 cores.

CMDLINE: python radical-pilot-agent.py -b 0 -c 2 -d 50 -j LOCAL -k MPIRUN -l FORK -m mongodb://ec2-184-72-89-141.compute-1.amazonaws.com:24242/ -n radicalpilot -p 53dfeeb71d2f566368b50f02 -s 53dfeeb51d2f566368b50f00 -t 5 -v 0.18

stderr ends with

x virtualenv-1.9/virtualenv_support/setuptools-0.6c11-py2.5.egg x virtualenv-1.9/virtualenv_support/setuptools-0.6c11-py2.6.egg x virtualenv-1.9/virtualenvsupport/setuptools-0.6c11-py2.7.egg warning: no files found matching '.html' under directory 'docs' warning: no previously-included files matching '_.txt' found under directory 'docs/_build' no previously-included directories found matching 'docs/_build/_sources' ./default_bootstrapper.sh: line 90: 25621 Killed: 9 $PIP_CMD ./default_bootstrapper.sh: line 421: 26343 Killed: 9 $AGENT_CMD

log is empty

marksantcroos commented 9 years ago

On 04 Aug 2014, at 23:27 , Daniel S. Katz notifications@github.com wrote:

note that this is a local test. should the network matter?

The network can/might matter for access to the MongoDB.

stdout ends with

################################################################################

Launching radical-pilot-agent for 2 cores.

CMDLINE: python radical-pilot-agent.py -b 0 -c 2 -d 50 -j LOCAL -k MPIRUN -l FORK -m mongodb://ec2-184-72-89-141.compute-1.amazonaws.com:24242/ -n radicalpilot -p 53dfeeb71d2f566368b50f02 -s 53dfeeb51d2f566368b50f00 -t 5 -v 0.18

So far so good.

stderr ends with

x virtualenv-1.9/virtualenv_support/setuptools-0.6c11-py2.5.egg x virtualenv-1.9/virtualenv_support/setuptools-0.6c11-py2.6.egg x virtualenv-1.9/virtualenv_support/setuptools-0.6c11-py2.7.egg warning: no files found matching '.html' under directory 'docs' warning: no previously-included files matching '.txt' found under directory 'docs/_build' no previously-included directories found matching 'docs/_build/_sources' ./default_bootstrapper.sh: line 90: 25621 Killed: 9 $PIP_CMD ./default_bootstrapper.sh: line 421: 26343 Killed: 9 $AGENT_CMD

log is empty

Hmmm, not much informative here.

Could you please retry with the following enabled? export RADICAL_PILOT_VERBOSE=DEBUG

Given that the bootstrap works fine, I will be mainly interested in the output of AGENT.LOG, or if there is something interesting on your console screen.

danielskatz commented 9 years ago

here's what's on my screen:

(sagenv)nsit-dhcp-250-241:examples dsk$ python getting_started_local.py
2014:08:05 13:39:39 radical.pilot.MainProcess: [INFO    ] radical.pilot version: 0.18
2014:08:05 13:39:40 radical.pilot.MainProcess: [INFO    ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/configs/das4.json
2014:08:05 13:39:40 radical.pilot.MainProcess: [INFO    ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/configs/epsrc.json
2014:08:05 13:39:40 radical.pilot.MainProcess: [INFO    ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/configs/futuregrid.json
2014:08:05 13:39:40 radical.pilot.MainProcess: [INFO    ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/configs/iu.json
2014:08:05 13:39:40 radical.pilot.MainProcess: [INFO    ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/configs/localhost.json
2014:08:05 13:39:40 radical.pilot.MainProcess: [INFO    ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/configs/lrz.json
2014:08:05 13:39:40 radical.pilot.MainProcess: [INFO    ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/configs/ncar.json
2014:08:05 13:39:41 radical.pilot.MainProcess: [INFO    ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/configs/xsede.json
2014:08:05 13:39:41 radical.pilot.MainProcess: [INFO    ] New Session created{'database_url': 'mongodb://ec2-184-72-89-141.compute-1.amazonaws.com:24242/', 'database_name': 'radicalpilot', 'last_reconnect': None, 'uid': '53e124ecd1969c73e56f0eb0', 'created': datetime.datetime(2014, 8, 5, 18, 39, 41, 127137)}.
2014:08:05 13:39:41 radical.pilot.MainProcess: [DEBUG   ] Worker thread (ID: Thread-1[4447342592]) for PilotManager 53e124edd1969c73e56f0eb1 started.
2014:08:05 13:39:41 radical.pilot.PilotLauncherWorker-1: [DEBUG   ] Connected to MongoDB. Serving requests for PilotManager 53e124edd1969c73e56f0eb1.
2014:08:05 13:39:41 radical.pilot.PilotLauncherWorker-1: [INFO    ] Launching ComputePilot {u'state': u'PendingLaunch', u'commands': [], u'description': {u'project': None, u'resource': u'localhost', u'queue': None, u'sandbox': None, u'cleanup': True, u'pilot_agent_priv': None, u'memory': None, u'cores': 2, u'runtime': 5}, u'sagajobid': None, u'started': None, u'cores_per_node': None, u'output_transfer_started': None, u'finished': None, u'submitted': datetime.datetime(2014, 8, 5, 18, 39, 41, 539000), u'output_transfer_finished': None, u'sandbox': u'file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/', u'pilotmanager': u'53e124edd1969c73e56f0eb1', u'unitmanager': None, u'heartbeat': None, u'wu_queue': [], u'statehistory': [{u'timestamp': datetime.datetime(2014, 8, 5, 18, 39, 41, 538000), u'state': u'PendingLaunch'}], u'input_transfer_started': None, u'_id': ObjectId('53e124edd1969c73e56f0eb2'), u'input_transfer_finished': None, u'nodes': None, u'log': []}
2014:08:05 13:39:41 radical.pilot.PilotLauncherWorker-1: [INFO    ] Using pilot agent /Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/agent/radical-pilot-agent-multicore.py
2014:08:05 13:39:41 radical.pilot.PilotLauncherWorker-1: [INFO    ] Using bootstrapper /Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/bootstrapper/default_bootstrapper.sh
2014:08:05 13:39:41 radical.pilot.PilotLauncherWorker-1: [DEBUG   ] Creating agent sandbox 'file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/'.
2014:08:05 13:39:41 radical.pilot.MainProcess: [DEBUG   ] Worker thread (ID: Thread-2[4451549184]) for UnitManager 53e124edd1969c73e56f0eb3 started.
2014:08:05 13:39:41 radical.pilot.MainProcess: [INFO    ] Loaded scheduler: DirectSubmissionScheduler.
2014:08:05 13:39:41 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Connected to MongoDB. Serving requests for UnitManager 53e124edd1969c73e56f0eb3.
2014:08:05 13:39:41 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Connected to MongoDB. Serving requests for UnitManager 53e124edd1969c73e56f0eb3.
2014:08:05 13:39:41 radical.pilot.OutputFileTransferWorker-2: [DEBUG   ] Connected to MongoDB. Serving requests for UnitManager 53e124edd1969c73e56f0eb3.
2014:08:05 13:39:41 radical.pilot.OutputFileTransferWorker-1: [DEBUG   ] Connected to MongoDB. Serving requests for UnitManager 53e124edd1969c73e56f0eb3.
2014:08:05 13:39:41 radical.pilot.MainProcess: [INFO    ] Scheduled ComputeUnits [] for execution on ComputePilot '53e124edd1969c73e56f0eb2'.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] ComputePilot '53e124edd1969c73e56f0eb2' state changed from 'PendingLaunch' to 'Launching'.
[Callback]: ComputePilot '53e124edd1969c73e56f0eb2' state changed to Launching.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb4' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb4' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb5' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb5' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb6' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb6' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb7' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb7' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb8' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb8' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb9' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb9' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eba' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eba' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebb' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebb' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebc' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebc' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebd' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebd' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebe' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebe' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebf' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebf' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec0' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec0' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec1' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec1' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec2' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec2' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec3' state changed from 'New' to 'PendingInputTransfer'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec3' state changed to PendingInputTransfer.
2014:08:05 13:39:42 radical.pilot.InputFileTransferWorker-2: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb4.
2014:08:05 13:39:42 radical.pilot.InputFileTransferWorker-1: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb5.
2014:08:05 13:39:43 radical.pilot.PilotLauncherWorker-1: [DEBUG   ] Copying bootstrapper 'file://localhost//Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/bootstrapper/default_bootstrapper.sh' to agent sandbox (file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/).
2014:08:05 13:39:43 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb4' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb4' state changed to TransferringInput.
2014:08:05 13:39:43 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb5' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb5' state changed to TransferringInput.
2014:08:05 13:39:44 radical.pilot.InputFileTransferWorker-2: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0eb4
2014:08:05 13:39:44 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb4
2014:08:05 13:39:44 radical.pilot.InputFileTransferWorker-1: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0eb5
2014:08:05 13:39:44 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb5
2014:08:05 13:39:44 radical.pilot.PilotLauncherWorker-1: [DEBUG   ] Copying agent 'file://localhost//Users/dsk/sagenv/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg/radical/pilot/agent/radical-pilot-agent-multicore.py' to agent sandbox (file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/).
2014:08:05 13:39:45 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb4
2014:08:05 13:39:45 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb5
2014:08:05 13:39:47 radical.pilot.InputFileTransferWorker-2: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb6.
2014:08:05 13:39:47 radical.pilot.InputFileTransferWorker-1: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb7.
2014:08:05 13:39:47 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb6' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb6' state changed to TransferringInput.
2014:08:05 13:39:47 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb7' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb7' state changed to TransferringInput.
2014:08:05 13:39:47 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb4' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb4' state changed to PendingExecution.
2014:08:05 13:39:47 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb5' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb5' state changed to PendingExecution.
2014:08:05 13:39:48 radical.pilot.PilotLauncherWorker-1: [DEBUG   ] Bootstrap command line: /bin/bash ['-l', '-c', '"chmod +x default_bootstrapper.sh && ./default_bootstrapper.sh -n radicalpilot -s 53e124ecd1969c73e56f0eb0 -p 53e124edd1969c73e56f0eb2 -t 5 -d 10 -c 2 -v 0.18 -m ec2-184-72-89-141.compute-1.amazonaws.com:24242/  -l FORK  -j LOCAL  -k MPIRUN  -x "']
2014:08:05 13:39:48 radical.pilot.PilotLauncherWorker-1: [DEBUG   ] Submitting SAGA job with description: {'Executable': '/bin/bash', 'WorkingDirectory': '/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2', 'WallTimeLimit': 5, 'Arguments': ['-l', '-c', '"chmod +x default_bootstrapper.sh && ./default_bootstrapper.sh -n radicalpilot -s 53e124ecd1969c73e56f0eb0 -p 53e124edd1969c73e56f0eb2 -t 5 -d 10 -c 2 -v 0.18 -m ec2-184-72-89-141.compute-1.amazonaws.com:24242/  -l FORK  -j LOCAL  -k MPIRUN  -x "'], 'Error': 'AGENT.STDERR', 'Output': 'AGENT.STDOUT', 'TotalCPUCount': 2}
2014:08:05 13:39:49 radical.pilot.InputFileTransferWorker-2: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0eb6
2014:08:05 13:39:49 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb6
2014:08:05 13:39:49 radical.pilot.InputFileTransferWorker-1: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0eb7
2014:08:05 13:39:49 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb7
2014:08:05 13:39:49 radical.pilot.PilotLauncherWorker-1: [DEBUG   ] SAGA job submitted with job id [fork://localhost]-[29764]
2014:08:05 13:39:49 radical.pilot.MainProcess: [INFO    ] ComputePilot '53e124edd1969c73e56f0eb2' state changed from 'Launching' to 'PendingActive'.
[Callback]: ComputePilot '53e124edd1969c73e56f0eb2' state changed to PendingActive.
2014:08:05 13:39:50 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb6
2014:08:05 13:39:50 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb7
2014:08:05 13:39:52 radical.pilot.InputFileTransferWorker-2: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb8.
2014:08:05 13:39:52 radical.pilot.InputFileTransferWorker-1: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb9.
2014:08:05 13:39:53 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb8' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb8' state changed to TransferringInput.
2014:08:05 13:39:53 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb9' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb9' state changed to TransferringInput.
2014:08:05 13:39:53 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb6' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb6' state changed to PendingExecution.
2014:08:05 13:39:53 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb7' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb7' state changed to PendingExecution.
2014:08:05 13:39:53 radical.pilot.InputFileTransferWorker-2: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0eb8
2014:08:05 13:39:53 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb8
2014:08:05 13:39:54 radical.pilot.InputFileTransferWorker-1: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0eb9
2014:08:05 13:39:54 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb9
2014:08:05 13:39:55 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb8
2014:08:05 13:39:55 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb9
2014:08:05 13:39:57 radical.pilot.InputFileTransferWorker-2: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eba.
2014:08:05 13:39:57 radical.pilot.InputFileTransferWorker-1: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebb.
2014:08:05 13:39:57 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eba' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eba' state changed to TransferringInput.
2014:08:05 13:39:57 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebb' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebb' state changed to TransferringInput.
2014:08:05 13:39:57 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb8' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb8' state changed to PendingExecution.
2014:08:05 13:39:57 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eb9' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eb9' state changed to PendingExecution.
2014:08:05 13:39:58 radical.pilot.InputFileTransferWorker-2: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0eba
2014:08:05 13:39:58 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eba
2014:08:05 13:39:58 radical.pilot.InputFileTransferWorker-1: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0ebb
2014:08:05 13:39:58 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebb
2014:08:05 13:40:00 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eba
2014:08:05 13:40:00 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebb
2014:08:05 13:40:02 radical.pilot.InputFileTransferWorker-2: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebc.
2014:08:05 13:40:02 radical.pilot.InputFileTransferWorker-1: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebd.
2014:08:05 13:40:02 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebc' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebc' state changed to TransferringInput.
2014:08:05 13:40:02 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebd' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebd' state changed to TransferringInput.
2014:08:05 13:40:02 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0eba' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0eba' state changed to PendingExecution.
2014:08:05 13:40:02 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebb' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebb' state changed to PendingExecution.
2014:08:05 13:40:03 radical.pilot.InputFileTransferWorker-2: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0ebc
2014:08:05 13:40:03 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebc
2014:08:05 13:40:04 radical.pilot.InputFileTransferWorker-1: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0ebd
2014:08:05 13:40:04 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebd
2014:08:05 13:40:05 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebc
2014:08:05 13:40:05 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebd
2014:08:05 13:40:07 radical.pilot.InputFileTransferWorker-1: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebe.
2014:08:05 13:40:07 radical.pilot.InputFileTransferWorker-2: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebf.
2014:08:05 13:40:07 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebe' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebe' state changed to TransferringInput.
2014:08:05 13:40:07 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebf' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebf' state changed to TransferringInput.
2014:08:05 13:40:07 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebd' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebd' state changed to PendingExecution.
2014:08:05 13:40:07 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebc' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebc' state changed to PendingExecution.
2014:08:05 13:40:09 radical.pilot.InputFileTransferWorker-2: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0ebf
2014:08:05 13:40:09 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebf
2014:08:05 13:40:09 radical.pilot.InputFileTransferWorker-1: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0ebe
2014:08:05 13:40:09 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebe
2014:08:05 13:40:11 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebf
2014:08:05 13:40:11 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebe
2014:08:05 13:40:11 radical.pilot.PilotLauncherWorker-1: [INFO    ] Performing periodical health check for 53e124edd1969c73e56f0eb2 (SAGA job id [fork://localhost]-[29764])
2014:08:05 13:40:12 radical.pilot.InputFileTransferWorker-2: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec0.
2014:08:05 13:40:13 radical.pilot.InputFileTransferWorker-1: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec1.
2014:08:05 13:40:13 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec0' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec0' state changed to TransferringInput.
2014:08:05 13:40:13 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec1' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec1' state changed to TransferringInput.
2014:08:05 13:40:13 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebf' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebf' state changed to PendingExecution.
2014:08:05 13:40:13 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ebe' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ebe' state changed to PendingExecution.
2014:08:05 13:40:14 radical.pilot.InputFileTransferWorker-2: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0ec0
2014:08:05 13:40:14 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec0
2014:08:05 13:40:14 radical.pilot.InputFileTransferWorker-1: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0ec1
2014:08:05 13:40:14 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec1
2014:08:05 13:40:16 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec0
2014:08:05 13:40:16 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec1
2014:08:05 13:40:18 radical.pilot.InputFileTransferWorker-2: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec2.
2014:08:05 13:40:18 radical.pilot.InputFileTransferWorker-1: [INFO    ] Creating ComputeUnit sandbox directory file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec3.
2014:08:05 13:40:18 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec2' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec2' state changed to TransferringInput.
2014:08:05 13:40:18 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec3' state changed from 'PendingInputTransfer' to 'TransferringInput'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec3' state changed to TransferringInput.
2014:08:05 13:40:18 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec0' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec0' state changed to PendingExecution.
2014:08:05 13:40:18 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec1' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec1' state changed to PendingExecution.
2014:08:05 13:40:20 radical.pilot.InputFileTransferWorker-2: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0ec2
2014:08:05 13:40:20 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec2
2014:08:05 13:40:20 radical.pilot.InputFileTransferWorker-1: [INFO    ] Processing input file transfers for ComputeUnit 53e124edd1969c73e56f0ec3
2014:08:05 13:40:20 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file1.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec3
2014:08:05 13:40:21 radical.pilot.InputFileTransferWorker-2: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec2
2014:08:05 13:40:21 radical.pilot.InputFileTransferWorker-1: [DEBUG   ] Transferring input file file://localhost//Users/dsk/Desktop/AIMES/radical.pilot/examples/file2.dat -> file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec3
2014:08:05 13:40:24 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec2' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec2' state changed to PendingExecution.
2014:08:05 13:40:24 radical.pilot.MainProcess: [INFO    ] RUN ComputeUnit '53e124edd1969c73e56f0ec3' state changed from 'TransferringInput' to 'PendingExecution'.
[Callback]: ComputeUnit '53e124edd1969c73e56f0ec3' state changed to PendingExecution.
2014:08:05 13:40:45 radical.pilot.PilotLauncherWorker-1: [INFO    ] Performing periodical health check for 53e124edd1969c73e56f0eb2 (SAGA job id [fork://localhost]-[29764])
2014:08:05 13:41:19 radical.pilot.PilotLauncherWorker-1: [INFO    ] Performing periodical health check for 53e124edd1969c73e56f0eb2 (SAGA job id [fork://localhost]-[29764])
2014:08:05 13:41:53 radical.pilot.PilotLauncherWorker-1: [INFO    ] Performing periodical health check for 53e124edd1969c73e56f0eb2 (SAGA job id [fork://localhost]-[29764])
2014:08:05 13:42:26 radical.pilot.PilotLauncherWorker-1: [INFO    ] Performing periodical health check for 53e124edd1969c73e56f0eb2 (SAGA job id [fork://localhost]-[29764])
2014:08:05 13:43:00 radical.pilot.PilotLauncherWorker-1: [INFO    ] Performing periodical health check for 53e124edd1969c73e56f0eb2 (SAGA job id [fork://localhost]-[29764])
2014:08:05 13:43:34 radical.pilot.PilotLauncherWorker-1: [INFO    ] Performing periodical health check for 53e124edd1969c73e56f0eb2 (SAGA job id [fork://localhost]-[29764])
2014:08:05 13:44:08 radical.pilot.PilotLauncherWorker-1: [INFO    ] Performing periodical health check for 53e124edd1969c73e56f0eb2 (SAGA job id [fork://localhost]-[29764])
2014:08:05 13:44:42 radical.pilot.PilotLauncherWorker-1: [INFO    ] Performing periodical health check for 53e124edd1969c73e56f0eb2 (SAGA job id [fork://localhost]-[29764])

danielskatz commented 9 years ago

here's AGENT.LOG

(sagenv)nsit-dhcp-250-241:pilot-53e124edd1969c73e56f0eb2 dsk$ cat AGENT.LOG
2014-08-05 13:45:18,992 - radical.pilot.agent - INFO - RADICAL-Pilot multi-core agent for package/API version 0.18
2014-08-05 13:45:19,514 - radical.pilot.agent - INFO - Using fork on localhost.
2014-08-05 13:45:19,521 - radical.pilot.agent - INFO - Detected 4 cores on localhost, using 2.
2014-08-05 13:45:19,524 - radical.pilot.agent - WARNING - No MPI launch command found for launch method: MPIRUN.
2014-08-05 13:45:19,524 - radical.pilot.agent - INFO - Discovered task launch command: 'None' and MPI launch command: 'None'.
2014-08-05 13:45:19,524 - radical.pilot.agent - INFO - Discovered execution environment: ['localhost']
2014-08-05 13:45:19,629 - radical.pilot.agent - INFO - Started up <ExecWorker(ExecWorker-1, started daemon)> serving nodes ['localhost']
2014-08-05 13:45:19,630 - radical.pilot.agent - INFO - Agent started. Database updated.
2014-08-05 13:45:19,632 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:19,705 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0eb4
2014-08-05 13:45:20,633 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:20,634 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:20,634 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:20,636 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb4/radical_pilot_cu_launch_script-PeX4wk.sh
2014-08-05 13:45:20,637 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0eb4 via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb4/radical_pilot_cu_launch_script-PeX4wk.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb4
2014-08-05 13:45:20,693 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0eb4 terminated with return code 0.
2014-08-05 13:45:20,754 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0eb5
2014-08-05 13:45:20,878 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb4/STDOUT to MongoDB as 53e12640d1969c76d4676f3d.
2014-08-05 13:45:20,948 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb4/STDERR to MongoDB as 53e12640d1969c76d4676f3f.
2014-08-05 13:45:20,971 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:20,972 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:20,972 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:20,973 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb5/radical_pilot_cu_launch_script-ifG_o7.sh
2014-08-05 13:45:20,973 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0eb5 via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb5/radical_pilot_cu_launch_script-ifG_o7.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb5
2014-08-05 13:45:21,001 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0eb5 terminated with return code 0.
2014-08-05 13:45:21,095 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb5/STDOUT to MongoDB as 53e12641d1969c76d4676f40.
2014-08-05 13:45:21,165 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb5/STDERR to MongoDB as 53e12641d1969c76d4676f42.
2014-08-05 13:45:21,188 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:21,802 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0eb6
2014-08-05 13:45:22,189 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:22,190 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:22,190 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:22,191 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb6/radical_pilot_cu_launch_script-jgFgRx.sh
2014-08-05 13:45:22,191 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0eb6 via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb6/radical_pilot_cu_launch_script-jgFgRx.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb6
2014-08-05 13:45:22,219 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0eb6 terminated with return code 0.
2014-08-05 13:45:22,312 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb6/STDOUT to MongoDB as 53e12642d1969c76d4676f43.
2014-08-05 13:45:22,383 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb6/STDERR to MongoDB as 53e12642d1969c76d4676f45.
2014-08-05 13:45:22,406 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:22,850 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0eb7
2014-08-05 13:45:23,408 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:23,408 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:23,409 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:23,409 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb7/radical_pilot_cu_launch_script-rVpty_.sh
2014-08-05 13:45:23,410 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0eb7 via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb7/radical_pilot_cu_launch_script-rVpty_.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb7
2014-08-05 13:45:23,441 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  X  -

2014-08-05 13:45:23,898 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0eb8
2014-08-05 13:45:24,443 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  X  -

2014-08-05 13:45:24,444 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:24,444 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 1
2014-08-05 13:45:24,445 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb8/radical_pilot_cu_launch_script-1nsu22.sh
2014-08-05 13:45:24,446 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0eb8 via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb8/radical_pilot_cu_launch_script-1nsu22.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb8
2014-08-05 13:45:24,474 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0eb7 terminated with return code 0.
2014-08-05 13:45:24,568 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb7/STDOUT to MongoDB as 53e12644d1969c76d4676f46.
2014-08-05 13:45:24,639 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb7/STDERR to MongoDB as 53e12644d1969c76d4676f48.
2014-08-05 13:45:24,639 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0eb8 terminated with return code 0.
2014-08-05 13:45:24,733 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb8/STDOUT to MongoDB as 53e12644d1969c76d4676f49.
2014-08-05 13:45:24,804 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb8/STDERR to MongoDB as 53e12644d1969c76d4676f4b.
2014-08-05 13:45:24,851 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:24,947 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0eb9
2014-08-05 13:45:25,852 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:25,853 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:25,854 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:25,855 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb9/radical_pilot_cu_launch_script-S7v8Um.sh
2014-08-05 13:45:25,855 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0eb9 via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb9/radical_pilot_cu_launch_script-S7v8Um.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb9
2014-08-05 13:45:25,883 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0eb9 terminated with return code 0.
2014-08-05 13:45:25,977 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb9/STDOUT to MongoDB as 53e12645d1969c76d4676f4c.
2014-08-05 13:45:25,996 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0eba
2014-08-05 13:45:26,048 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eb9/STDERR to MongoDB as 53e12645d1969c76d4676f4e.
2014-08-05 13:45:26,072 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:26,073 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:26,073 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:26,074 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eba/radical_pilot_cu_launch_script-tWgRYz.sh
2014-08-05 13:45:26,075 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0eba via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eba/radical_pilot_cu_launch_script-tWgRYz.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eba
2014-08-05 13:45:26,103 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0eba terminated with return code 0.
2014-08-05 13:45:26,195 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eba/STDOUT to MongoDB as 53e12646d1969c76d4676f4f.
2014-08-05 13:45:26,265 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0eba/STDERR to MongoDB as 53e12646d1969c76d4676f51.
2014-08-05 13:45:26,289 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:27,044 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0ebb
2014-08-05 13:45:27,291 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:27,292 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:27,292 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:27,293 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebb/radical_pilot_cu_launch_script-xiKwr_.sh
2014-08-05 13:45:27,294 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0ebb via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebb/radical_pilot_cu_launch_script-xiKwr_.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebb
2014-08-05 13:45:27,321 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0ebb terminated with return code 0.
2014-08-05 13:45:27,415 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebb/STDOUT to MongoDB as 53e12647d1969c76d4676f52.
2014-08-05 13:45:27,486 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebb/STDERR to MongoDB as 53e12647d1969c76d4676f54.
2014-08-05 13:45:27,510 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:28,092 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0ebd
2014-08-05 13:45:28,511 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:28,512 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:28,512 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:28,513 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebd/radical_pilot_cu_launch_script-jhR2YJ.sh
2014-08-05 13:45:28,513 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0ebd via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebd/radical_pilot_cu_launch_script-jhR2YJ.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebd
2014-08-05 13:45:28,540 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0ebd terminated with return code 0.
2014-08-05 13:45:28,633 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebd/STDOUT to MongoDB as 53e12648d1969c76d4676f55.
2014-08-05 13:45:28,703 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebd/STDERR to MongoDB as 53e12648d1969c76d4676f57.
2014-08-05 13:45:28,727 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:29,141 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0ebc
2014-08-05 13:45:29,728 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:29,729 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:29,729 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:29,730 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebc/radical_pilot_cu_launch_script-NpRTUp.sh
2014-08-05 13:45:29,730 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0ebc via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebc/radical_pilot_cu_launch_script-NpRTUp.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebc
2014-08-05 13:45:29,758 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0ebc terminated with return code 0.
2014-08-05 13:45:29,853 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebc/STDOUT to MongoDB as 53e12649d1969c76d4676f58.
2014-08-05 13:45:29,923 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebc/STDERR to MongoDB as 53e12649d1969c76d4676f5a.
2014-08-05 13:45:29,947 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:30,190 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0ebf
2014-08-05 13:45:30,949 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:30,949 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:30,949 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:30,950 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebf/radical_pilot_cu_launch_script-iZN8yb.sh
2014-08-05 13:45:30,950 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0ebf via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebf/radical_pilot_cu_launch_script-iZN8yb.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebf
2014-08-05 13:45:30,977 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0ebf terminated with return code 0.
2014-08-05 13:45:31,071 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebf/STDOUT to MongoDB as 53e1264ad1969c76d4676f5b.
2014-08-05 13:45:31,143 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebf/STDERR to MongoDB as 53e1264bd1969c76d4676f5d.
2014-08-05 13:45:31,166 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:31,238 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0ebe
2014-08-05 13:45:32,168 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:32,168 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:32,169 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:32,170 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebe/radical_pilot_cu_launch_script-gWflow.sh
2014-08-05 13:45:32,171 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0ebe via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebe/radical_pilot_cu_launch_script-gWflow.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebe
2014-08-05 13:45:32,198 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0ebe terminated with return code 0.
2014-08-05 13:45:32,287 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0ec0
2014-08-05 13:45:32,292 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebe/STDOUT to MongoDB as 53e1264cd1969c76d4676f5e.
2014-08-05 13:45:32,363 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ebe/STDERR to MongoDB as 53e1264cd1969c76d4676f60.
2014-08-05 13:45:32,387 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:32,387 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:32,388 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:32,388 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec0/radical_pilot_cu_launch_script-Hb0m4r.sh
2014-08-05 13:45:32,389 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0ec0 via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec0/radical_pilot_cu_launch_script-Hb0m4r.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec0
2014-08-05 13:45:32,418 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  X  -

2014-08-05 13:45:33,336 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0ec1
2014-08-05 13:45:33,420 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  X  -

2014-08-05 13:45:33,420 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:33,421 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 1
2014-08-05 13:45:33,422 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec1/radical_pilot_cu_launch_script-37Se8K.sh
2014-08-05 13:45:33,423 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0ec1 via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec1/radical_pilot_cu_launch_script-37Se8K.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec1
2014-08-05 13:45:33,451 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0ec0 terminated with return code 0.
2014-08-05 13:45:33,545 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec0/STDOUT to MongoDB as 53e1264dd1969c76d4676f61.
2014-08-05 13:45:33,615 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec0/STDERR to MongoDB as 53e1264dd1969c76d4676f63.
2014-08-05 13:45:33,616 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0ec1 terminated with return code 0.
2014-08-05 13:45:33,709 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec1/STDOUT to MongoDB as 53e1264dd1969c76d4676f64.
2014-08-05 13:45:33,780 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec1/STDERR to MongoDB as 53e1264dd1969c76d4676f66.
2014-08-05 13:45:33,827 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:34,385 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0ec2
2014-08-05 13:45:34,828 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:34,829 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:34,829 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:34,831 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec2/radical_pilot_cu_launch_script-L4q36p.sh
2014-08-05 13:45:34,832 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0ec2 via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec2/radical_pilot_cu_launch_script-L4q36p.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec2
2014-08-05 13:45:34,860 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0ec2 terminated with return code 0.
2014-08-05 13:45:34,953 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec2/STDOUT to MongoDB as 53e1264ed1969c76d4676f67.
2014-08-05 13:45:35,024 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec2/STDERR to MongoDB as 53e1264ed1969c76d4676f69.
2014-08-05 13:45:35,048 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:35,435 - radical.pilot.agent - INFO - Found new tasks in pilot queue: 53e124edd1969c73e56f0ec3
2014-08-05 13:45:36,049 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:36,050 - radical.pilot.agent - DEBUG - Launching task with LOCAL (None).
2014-08-05 13:45:36,050 - radical.pilot.agent - INFO - Node localhost satisfies 1 cores at offset 0
2014-08-05 13:45:36,051 - radical.pilot.agent - DEBUG - Created launch_script: /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec3/radical_pilot_cu_launch_script-JPQtuU.sh
2014-08-05 13:45:36,052 - radical.pilot.agent - INFO - Launching task 53e124edd1969c73e56f0ec3 via /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec3/radical_pilot_cu_launch_script-JPQtuU.sh in /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec3
2014-08-05 13:45:36,080 - radical.pilot.agent - INFO - Task 53e124edd1969c73e56f0ec3 terminated with return code 0.
2014-08-05 13:45:36,174 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec3/STDOUT to MongoDB as 53e12650d1969c76d4676f6a.
2014-08-05 13:45:36,245 - radical.pilot.agent - INFO - Uploaded /Users/dsk/radical.pilot.sandbox/pilot-53e124edd1969c73e56f0eb2/unit-53e124edd1969c73e56f0ec3/STDERR to MongoDB as 53e12650d1969c76d4676f6c.
2014-08-05 13:45:36,268 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:37,270 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:38,272 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:39,273 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:40,275 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:41,277 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:42,279 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:43,280 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:44,281 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:45,283 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:46,285 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:47,287 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:48,288 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:49,289 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:50,291 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:51,293 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:52,294 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:53,295 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:54,297 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:55,299 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:56,300 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:57,302 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:58,303 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:45:59,304 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:00,306 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:01,308 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:02,310 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:03,311 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:04,312 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:05,314 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:06,314 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:07,315 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:08,317 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:09,319 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:10,320 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:11,322 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:12,323 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:13,324 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:14,325 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:15,327 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:16,328 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:17,329 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:18,331 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:19,333 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:20,334 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:21,336 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:22,338 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:23,339 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:24,340 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:25,342 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:26,344 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:27,345 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:28,346 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:29,348 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:30,349 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:31,351 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:32,353 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:33,355 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:34,356 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:35,358 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:36,360 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:37,361 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:38,362 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:39,364 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:40,366 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:41,368 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:42,369 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:43,370 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:44,372 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:45,373 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:46,375 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:47,377 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:48,378 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:49,379 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:50,381 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:51,382 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:52,384 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:53,386 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:54,387 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:55,389 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:56,390 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:57,392 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:58,393 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:46:59,394 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:47:00,396 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

2014-08-05 13:47:01,397 - radical.pilot.agent - DEBUG - Slot status:
localhost               :  -  -

andre-merzky commented 9 years ago

Your units are run and finish successful -- but the client side gets never notified about the state change it seems.

Its interesting, and a little scary, that both you and Francis hit so many issues with the simple examples which we considered stable :/

I am not sure how much time you are willing to spend on this, to help us to debug. If you would not mind, could you please let me know:

the DB URL you have been using
the session ID (if you have that). If you don't, can you please try again, and after creating the session call print session.uid (if session is the session's variable name in your script)? With that ID I can check the state of the run in the database.
if you rerun, please use RADICAL_PILOT_VERBOSE=DEBUG

Thanks, Andre.

danielskatz commented 9 years ago

DB: mongodb://ec2-184-72-89-141.compute-1.amazonaws.com:24242/ session ID - going from the directory name, assuming this is what you want:pilot-53e124edd1969c73e56f0eb2

let me know if I need to do something else, very specifically what I should do :)

andre-merzky commented 9 years ago

Thanks! The pilot ID suffices (we can get the session ID from there).

But this is quite strange: I find the session all right I think -- but most of its content seems gone. That is very confusing.

Could you please try again, and explicitly print the session ID (just to make sure I am not doing this mapping wrong)? Please, also run with `RADICAL_PILOT_VERBOSE=DEBUG, and replace

session.close()

with

session.close (remove=False)

in the script. That should make sure the session survives (although in your case we are not really getting this far, and the code should already contain this).

marksantcroos commented 9 years ago

Dan uses the new DB port, in that one you used all seems fine. Doesn't explain why the client does not become aware of things completing though ...

Have to think a bit about this one, will try to reproduce.

On 05 Aug 2014, at 21:39 , Andre Merzky notifications@github.com wrote:

Thanks! The pilot ID suffices (we can get the session ID from there).

But this is quite strange: I find the session all right I think -- but most of its content seems gone. That is very confusing.

Could you please try again, and explicitly print the session ID (just to make sure I am not doing this mapping wrong)? Please, also run with `RADICAL_PILOT_VERBOSE=DEBUG, and replace

session.close()

with

session.close (remove=False)

in the script. That should make sure the session survives (although in your case we are not really getting this far, and the code should already contain this).

— Reply to this email directly or view it on GitHub.

andre-merzky commented 9 years ago

Thanks Mark, sorry for the confusion. Indeed strange though that all finishes nicely, but the client does not stop :/

andre-merzky commented 9 years ago

Hi Dan,

I heard you were back -- and still hung on this item, right? I still could not reproduce the problem I'm afraid -- but we fixed a number of related issues in a different branch (which is the release candidate for our upcoming milestone). Would you mind giving the experimental branch a try? The following set of commands seem to work out of the box for me, on different machines:

$  cd /tmp/
$  rm -rf ve
$  virtualenv ve
$  source ve/bin/activate
$  git clone git@github.com:radical-cybertools/radical.pilot.git
$  cd radical.pilot/
$  git co experimental
$  pip install --upgrade .
$  vi examples/getting_started_local.py
   # check what this does, adapt size of pilots and number of units as needed
$  hostname > file1.dat
$  date > file2.dat
$  python examples/getting_started_local.py
$  vi examples/getting_started_remote.py
   # change target host to a host with access, like india.futuregrid.org
$  python examples/getting_started_remote.py

Thanks, Andre.

danielskatz commented 9 years ago

This still seems to hang:

(sagenv)nsit-dhcp-250-241:examples dsk$ hostname > file1.dat (sagenv)nsit-dhcp-250-241:examples dsk$ date > file2.dat (sagenv)nsit-dhcp-250-241:examples dsk$ python getting_started_local.py session id: 53f49cf94c425fec95d25684 [Callback]: ComputePilot '53f49cfb4c425fec95d25686' state: Launching. [Callback]: ComputePilot '53f49cfb4c425fec95d25687' state: Launching. [Callback]: ComputePilot '53f49cfb4c425fec95d25686' state: PendingActive. [Callback]: ComputePilot '53f49cfb4c425fec95d25687' state: PendingActive.

Here's a version with debugging on:

(sagenv)nsit-dhcp-250-241:examples dsk$ export RADICAL_PILOT_VERBOSE=DEBUG (sagenv)nsit-dhcp-250-241:examples dsk$ (sagenv)nsit-dhcp-250-241:examples dsk$ hostname > file1.dat (sagenv)nsit-dhcp-250-241:examples dsk$ date > file2.dat (sagenv)nsit-dhcp-250-241:examples dsk$ python getting_started_local.py 2014:08:20 09:15:37 radical.pilot.MainProcess: [INFO ] radical.pilot version: 0.18 2014:08:20 09:15:38 radical.pilot.MainProcess: [INFO ] using database url mongodb://ec2-184-72-89-141.compute-1.amazonaws.com:24242/ 2014:08:20 09:15:38 radical.pilot.MainProcess: [INFO ] using database name radicalpilot 2014:08:20 09:15:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/configs/das4.json 2014:08:20 09:15:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/configs/epsrc.json 2014:08:20 09:15:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/configs/futuregrid.json 2014:08:20 09:15:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/configs/iu.json 2014:08:20 09:15:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/configs/localhost.json 2014:08:20 09:15:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/configs/lrz.json 2014:08:20 09:15:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/configs/ncar.json 2014:08:20 09:15:38 radical.pilot.MainProcess: [INFO ] Loaded resource configurations from /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/configs/xsede.json 2014:08:20 09:15:39 radical.pilot.MainProcess: [INFO ] New Session created{'database_url': 'mongodb://ec2-184-72-89-141.compute-1.amazonaws.com:24242/', 'database_name': 'radicalpilot', 'last_reconnect': None, 'uid': '53f49f7a4c425fee29cce04e', 'created': datetime.datetime(2014, 8, 20, 13, 15, 38, 851648)}. session id: 53f49f7a4c425fee29cce04e 2014:08:20 09:15:39 radical.pilot.MainProcess: [DEBUG ] Worker thread (ID: Thread-1[4500910080]) for PilotManager 53f49f7b4c425fee29cce04f started. 2014:08:20 09:15:39 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for PilotManager 53f49f7b4c425fee29cce04f. 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] Launching ComputePilot {u'state': u'PendingLaunch', u'commands': [], u'description': {u'project': None, u'resource': u'localhost', u'queue': None, u'sandbox': None, u'cleanup': True, u'pilot_agent_priv': None, u'memory': None, u'cores': 4, u'runtime': 10}, u'sagajobid': None, u'started': None, u'cores_per_node': None, u'output_transfer_started': None, u'finished': None, u'submitted': datetime.datetime(2014, 8, 20, 13, 15, 39, 775000), u'output_transfer_finished': None, u'sandbox': u'file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53f49f7b4c425fee29cce050/', u'pilotmanager': u'53f49f7b4c425fee29cce04f', u'unitmanager': None, u'heartbeat': None, u'statehistory': [{u'timestamp': datetime.datetime(2014, 8, 20, 13, 15, 39, 775000), u'state': u'PendingLaunch'}], u'input_transfer_started': None, u'_id': ObjectId('53f49f7b4c425fee29cce050'), u'input_transfer_finished': None, u'nodes': None, u'log': []} 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] Starting InputFileTransferWorker 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] Starting InputFileTransferWorker 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Worker thread (ID: Thread-3[4521943040]) for UnitManager 53f49f7c4c425fee29cce052 started. 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] Using pilot agent /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/agent/radical-pilot-agent-multicore.py 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] Loaded scheduler: BackfillingScheduler. 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] Using bootstrapper /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Creating agent sandbox 'file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53f49f7b4c425fee29cce050/'. 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 53f49f7c4c425fee29cce052. 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 53f49f7c4c425fee29cce052. 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 53f49f7c4c425fee29cce052. 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Connected to MongoDB. Serving requests for UnitManager 53f49f7c4c425fee29cce052. 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file1.dat' into dict '{'action': 'Transfer', 'source': 'file1.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file1.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] Converting string 'file2.dat' into dict '{'action': 'Transfer', 'source': 'file2.dat', 'flags': ['CreateParents', 'SkipFailed'], 'target': 'file2.dat', 'priority': 0}' 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce057 remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce059 remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce056 remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce05b remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce05d remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce062 remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce05f remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce061 remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce053 remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce05c remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce05a remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce054 remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce055 remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce058 remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce060 remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] unit 53f49f7c4c425fee29cce05e remains unscheduled 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] skipping empty unit schedule 2014:08:20 09:15:40 radical.pilot.MainProcess: [INFO ] ComputePilot '53f49f7b4c425fee29cce050' state changed from 'PendingLaunch' to 'Launching'. [Callback]: ComputePilot '53f49f7b4c425fee29cce050' state: Launching. 2014:08:20 09:15:40 radical.pilot.MainProcess: [DEBUG ] [SchedulerCallback]: ComputePilot 53f49f7b4c425fee29cce050 changed to Launching 2014:08:20 09:15:42 radical.pilot.MainProcess: [DEBUG ] Copying bootstrapper 'file://localhost//Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh' to agent sandbox (file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53f49f7b4c425fee29cce050/). 2014:08:20 09:15:43 radical.pilot.MainProcess: [DEBUG ] Copying agent 'file://localhost//Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/agent/radical-pilot-agent-multicore.py' to agent sandbox (file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53f49f7b4c425fee29cce050/). 2014:08:20 09:15:46 radical.pilot.MainProcess: [INFO ] request cleanup for pilot 53f49f7b4c425fee29cce050 2014:08:20 09:15:46 radical.pilot.MainProcess: [DEBUG ] Bootstrap command line: /bin/bash ['-l', 'default_bootstrapper.sh', '-n radicalpilot -s 53f49f7a4c425fee29cce04e -p 53f49f7b4c425fee29cce050 -t 10 -d 10 -c 4 -v 0.18 -m ec2-184-72-89-141.compute-1.amazonaws.com:24242 -l FORK -j LOCAL -k MPIRUN -x luve'] 2014:08:20 09:15:46 radical.pilot.MainProcess: [DEBUG ] Submitting SAGA job with description: {'Executable': '/bin/bash', 'WorkingDirectory': '/Users/dsk/radical.pilot.sandbox/pilot-53f49f7b4c425fee29cce050', 'WallTimeLimit': 10, 'Arguments': ['-l', 'default_bootstrapper.sh', '-n radicalpilot -s 53f49f7a4c425fee29cce04e -p 53f49f7b4c425fee29cce050 -t 10 -d 10 -c 4 -v 0.18 -m ec2-184-72-89-141.compute-1.amazonaws.com:24242 -l FORK -j LOCAL -k MPIRUN -x luve'], 'Error': 'AGENT.STDERR', 'Output': 'AGENT.STDOUT', 'TotalCPUCount': 4} 2014:08:20 09:15:46 radical.pilot.MainProcess: [DEBUG ] SAGA job submitted with job id [fork://localhost]-[61026] 2014:08:20 09:15:47 radical.pilot.MainProcess: [INFO ] Launching ComputePilot {u'state': u'PendingLaunch', u'commands': [], u'description': {u'project': None, u'resource': u'localhost', u'queue': None, u'sandbox': None, u'cleanup': True, u'pilot_agent_priv': None, u'memory': None, u'cores': 4, u'runtime': 10}, u'sagajobid': None, u'started': None, u'cores_per_node': None, u'output_transfer_started': None, u'sandbox': u'file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53f49f7b4c425fee29cce051/', u'submitted': datetime.datetime(2014, 8, 20, 13, 15, 39, 899000), u'output_transfer_finished': None, u'finished': None, u'pilotmanager': u'53f49f7b4c425fee29cce04f', u'unitmanager': u'53f49f7c4c425fee29cce052', u'statehistory': [{u'timestamp': datetime.datetime(2014, 8, 20, 13, 15, 39, 898000), u'state': u'PendingLaunch'}], u'heartbeat': None, u'input_transfer_started': None, u'_id': ObjectId('53f49f7b4c425fee29cce051'), u'input_transfer_finished': None, u'nodes': None, u'log': []} 2014:08:20 09:15:47 radical.pilot.MainProcess: [INFO ] Using pilot agent /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/agent/radical-pilot-agent-multicore.py 2014:08:20 09:15:47 radical.pilot.MainProcess: [INFO ] Using bootstrapper /Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh 2014:08:20 09:15:47 radical.pilot.MainProcess: [DEBUG ] Creating agent sandbox 'file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53f49f7b4c425fee29cce051/'. 2014:08:20 09:15:48 radical.pilot.MainProcess: [INFO ] ComputePilot '53f49f7b4c425fee29cce051' state changed from 'PendingLaunch' to 'Launching'. [Callback]: ComputePilot '53f49f7b4c425fee29cce051' state: Launching. 2014:08:20 09:15:48 radical.pilot.MainProcess: [DEBUG ] [SchedulerCallback]: ComputePilot 53f49f7b4c425fee29cce051 changed to Launching 2014:08:20 09:15:48 radical.pilot.MainProcess: [INFO ] ComputePilot '53f49f7b4c425fee29cce050' state changed from 'Launching' to 'PendingActive'. [Callback]: ComputePilot '53f49f7b4c425fee29cce050' state: PendingActive. 2014:08:20 09:15:48 radical.pilot.MainProcess: [DEBUG ] [SchedulerCallback]: ComputePilot 53f49f7b4c425fee29cce050 changed to PendingActive 2014:08:20 09:15:48 radical.pilot.MainProcess: [DEBUG ] Copying bootstrapper 'file://localhost//Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh' to agent sandbox (file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53f49f7b4c425fee29cce051/). 2014:08:20 09:15:50 radical.pilot.MainProcess: [DEBUG ] Copying agent 'file://localhost//Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/agent/radical-pilot-agent-multicore.py' to agent sandbox (file://localhost/Users/dsk/radical.pilot.sandbox/pilot-53f49f7b4c425fee29cce051/). 2014:08:20 09:15:54 radical.pilot.MainProcess: [INFO ] request cleanup for pilot 53f49f7b4c425fee29cce051 2014:08:20 09:15:54 radical.pilot.MainProcess: [DEBUG ] Bootstrap command line: /bin/bash ['-l', 'default_bootstrapper.sh', '-n radicalpilot -s 53f49f7a4c425fee29cce04e -p 53f49f7b4c425fee29cce051 -t 10 -d 10 -c 4 -v 0.18 -m ec2-184-72-89-141.compute-1.amazonaws.com:24242 -l FORK -j LOCAL -k MPIRUN -x luve'] 2014:08:20 09:15:54 radical.pilot.MainProcess: [DEBUG ] Submitting SAGA job with description: {'Executable': '/bin/bash', 'WorkingDirectory': '/Users/dsk/radical.pilot.sandbox/pilot-53f49f7b4c425fee29cce051', 'WallTimeLimit': 10, 'Arguments': ['-l', 'default_bootstrapper.sh', '-n radicalpilot -s 53f49f7a4c425fee29cce04e -p 53f49f7b4c425fee29cce051 -t 10 -d 10 -c 4 -v 0.18 -m ec2-184-72-89-141.compute-1.amazonaws.com:24242 -l FORK -j LOCAL -k MPIRUN -x luve'], 'Error': 'AGENT.STDERR', 'Output': 'AGENT.STDOUT', 'TotalCPUCount': 4} 2014:08:20 09:15:54 radical.pilot.MainProcess: [DEBUG ] SAGA job submitted with job id [fork://localhost]-[61117] 2014:08:20 09:15:55 radical.pilot.MainProcess: [INFO ] ComputePilot '53f49f7b4c425fee29cce051' state changed from 'Launching' to 'PendingActive'. [Callback]: ComputePilot '53f49f7b4c425fee29cce051' state: PendingActive. 2014:08:20 09:15:55 radical.pilot.MainProcess: [DEBUG ] [SchedulerCallback]: ComputePilot 53f49f7b4c425fee29cce051 changed to PendingActive 2014:08:20 09:16:40 radical.pilot.MainProcess: [INFO ] Performing periodical health check for 53f49f7b4c425fee29cce050 (SAGA job id [fork://localhost]-[61026]) 2014:08:20 09:16:43 radical.pilot.MainProcess: [INFO ] pilot 53f49f7b4c425fee29cce050 alive and well (0) 2014:08:20 09:16:43 radical.pilot.MainProcess: [INFO ] Performing periodical health check for 53f49f7b4c425fee29cce051 (SAGA job id [fork://localhost]-[61117]) 2014:08:20 09:16:44 radical.pilot.MainProcess: [INFO ] pilot 53f49f7b4c425fee29cce051 alive and well (0) 2014:08:20 09:17:40 radical.pilot.MainProcess: [INFO ] Performing periodical health check for 53f49f7b4c425fee29cce050 (SAGA job id [fork://localhost]-[61026]) 2014:08:20 09:17:41 radical.pilot.MainProcess: [INFO ] pilot 53f49f7b4c425fee29cce050 alive and well (0) 2014:08:20 09:17:41 radical.pilot.MainProcess: [INFO ] Performing periodical health check for 53f49f7b4c425fee29cce051 (SAGA job id [fork://localhost]-[61117]) 2014:08:20 09:17:42 radical.pilot.MainProcess: [INFO ] pilot 53f49f7b4c425fee29cce051 alive and well (0) ^CTraceback (most recent call last): File "getting_started_local.py", line 147, in umgr.wait_units() File "/Users/dsk/sagenv/lib/python2.7/site-packages/radical/pilot/unit_manager.py", line 653, in wait_units time.sleep (1) KeyboardInterrupt

andre-merzky commented 9 years ago

Dan, a quick note: I do not really know what to make of the problem, and can't reproduce it anywhere. We will have to add some more fine-grained debug messages to track this down it seems -- from the current debug log its not obvious were the state information gets lost (not to speak of the why)...

marksantcroos commented 9 years ago

Dan, did you have any luck with later versions of RP or you haven't tried anymore?

danielskatz commented 9 years ago

I didn't try again, sorry.

On Apr 22, 2015, at 17:51, Mark Santcroos notifications@github.com wrote:

Dan, did you have any luck with later versions of RP or you haven't tried anymore?

— Reply to this email directly or view it on GitHub.

andre-merzky commented 9 years ago

No problem, Dan. The code is now sufficiently different though that we would like to close this ticket, and rather start afresh whenever we can convince you to have another look at RP, if that's ok with you? Thanks!

danielskatz commented 9 years ago

Yes please.

On Apr 23, 2015, at 03:51, Andre Merzky notifications@github.com wrote:

No problem, Dan. The code is now sufficiently different though that we would like to close this ticket, and rather start afresh whenever we can convince you to have another look at RP, if that's ok with you? Thanks!

— Reply to this email directly or view it on GitHub.

marksantcroos commented 9 years ago

Thanks

radical-cybertools / radical.pilot

getting_started_local.py hangs #276

Launching radical-pilot-agent for 2 cores.

CMDLINE: python radical-pilot-agent.py -b 0 -c 2 -d 50 -j LOCAL -k MPIRUN -l FORK -m mongodb://ec2-184-72-89-141.compute-1.amazonaws.com:24242/ -n radicalpilot -p 53dfeeb71d2f566368b50f02 -s 53dfeeb51d2f566368b50f00 -t 5 -v 0.18