search

radical-cybertools / ExTASY

MDEnsemble
Other
1 stars 1 forks source link

Amber/Coco workflow linux/stampede fails #65

Closed ashkurti closed 9 years ago

ashkurti commented 9 years ago

I get the following error:

[ExTASY-toolsOct1] ardita@poirot 151% extasy --RPconfig stampede.rcfg --Kconfig cocoamber.wcfg Session UID: 542c2d98f8cdba30bd6bf5a2 Pilot UID : 542c2d9bf8cdba30bd6bf5a4 Loading kernel configurations from /users/ardita/ExTASY-toolsOct1/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/gromacs.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct1/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/amber.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct1/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/mmpbsa.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct1/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/test.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct1/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/coco.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct1/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/lsdmap.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct1/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/sleep.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct1/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/namd.json Cycle 0 Starting Simulation Traceback (most recent call last): File "/users/ardita/.local/bin/extasy", line 9, in load_entry_point('radical.ensemblemd.extasy==0.1', 'console_scripts', 'extasy')() File "/users/ardita/.local/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/runme.py", line 130, in main Simulator(umgr,RPconfig,Kconfig,i) File "/users/ardita/.local/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Amber/simulator.py", line 33, in Simulator for i in range(Kconfig.nreps): AttributeError: 'module' object has no attribute 'nreps'

vivek-bala commented 9 years ago

Please reinstall extasy from devel.

ashkurti commented 9 years ago

I followed the new instructions at https://github.com/radical-cybertools/ExTASY/blob/devel/README.md and installed extasy using: pip install --upgrade git+https://github.com/radical-cybertools/ExTASY.git@devel#egg=radical.ensemblemd.extasy I think this refers to devel doesn't it?

vivek-bala commented 9 years ago

Hmmm, could you try with a new virtualenv. Because line 33 in simulator.py should say for i in range(Kconfig.num_CUs): as can be seen in https://github.com/radical-cybertools/ExTASY/blob/devel/src/radical/ensemblemd/extasy/bin/Simulator/Amber/simulator.py.

ashkurti commented 9 years ago

I used a new virtualenv to follow the steps at https://github.com/radical-cybertools/ExTASY/blob/devel/README.md. I can use a new virtualenv, but should then use again the same instructions as at https://github.com/radical-cybertools/ExTASY/blob/devel/README.md. I do not get what should I do different with respect to before...

vivek-bala commented 9 years ago

Yes, the same instructions but a new virtualenv to get the updated devel extasy code.

ashkurti commented 9 years ago

Ok. I will retry now.

ashkurti commented 9 years ago

Ok, following the instructions as a user (without modifying any PATH or PYTHONPATH environment variables) I get the following error now: [ExTASY-toolsOct1] ardita@poirot 114% extasy --RPconfig stampede.rcfg --Kconfig cocoamber.wcfg Session UID: 542c37e5f8cdba34d65adc89 Pilot UID : 542c37e8f8cdba34d65adc8b Traceback (most recent call last): File "/users/ardita/.local/bin/extasy", line 9, in load_entry_point('radical.ensemblemd.extasy==0.1', 'console_scripts', 'extasy')() File "/users/ardita/.local/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/runme.py", line 117, in main from Simulator.Amber.simulator import Simulator File "/users/ardita/.local/lib/python2.7/site-packages/radical.ensemblemd.extasy-0.1-py2.7.egg/radical/ensemblemd/extasy/bin/Simulator/Amber/simulator.py", line 4, in from radical.ensemblemd.mdkernels import MDTaskDescription ImportError: No module named mdkernels

vivek-bala commented 9 years ago

I think you might have missed installation of mdkernels, step 2 : pip install --upgrade git+https://github.com/radical-cybertools/radical.ensemblemd.mdkernels.git@master#egg=radical.ensemblemd.mdkernels

vivek-bala commented 9 years ago

I've made a few commits to devel and changes to mdkernels in the last few minutes to get Amber-CoCo to work with .ncdf files. Please re-install in a fresh virtual environment to test.

ashkurti commented 9 years ago

I think my problem had to do with an update of the PATH variable. I have restarted testing in a new virtual environment now.

ashkurti commented 9 years ago

Now I get the following error: Starting Analysis Submitting COCO Compute Unit [Callback]: ComputeUnit '542d6a76f8cdba26b620157d' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6a76f8cdba26b620157d' state changed to StagingInput. [Callback]: ComputeUnit '542d6a76f8cdba26b620157d' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

ashkurti commented 9 years ago

I am launching the workflow from the "/users/ardita/coam-on-stampede" directory in case it might help when going through the details of the error above.

vivek-bala commented 9 years ago

Hmmm.. Which cycle is this ? Did the previous simulation finish successfully ? Could you paste the entire output please.

ashkurti commented 9 years ago

Yes, sure. This is cycle 1. I can find in the launching directory the following files: [ExTASY-toolsOct2] ardita@poirot 131% ls cocoamber.wcfgc md_0_1.ncdf md_0_3.ncdf md_0_5.ncdf md_0_7.ncdf min.in penta.top stampede.rcfgc cocoamber.wcfg md_0_0.ncdf md_0_2.ncdf md_0_4.ncdf md_0_6.ncdf mdshort.in penta.crd stampede.rcfg

I relaunched the workflow again and the whole output is the following: [ExTASY-toolsOct2] ardita@poirot 129% extasy --RPconfig stampede.rcfg --Kconfig cocoamber.wcfg Session UID: 542d6ea5f8cdba3e87873fd1 Pilot UID : 542d6ea8f8cdba3e87873fd3 [Callback]: ComputePilot '542d6ea8f8cdba3e87873fd3' state changed to Launching. Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/gromacs.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/amber.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/mmpbsa.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/test.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/coco.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/lsdmap.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/sleep.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/namd.json Cycle 0 Starting Simulation [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd6' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd9' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdb' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd5' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd7' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fda' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdc' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd8' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd6' state changed to StagingInput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd9' state changed to StagingInput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdb' state changed to StagingInput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd6' state changed to PendingExecution. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd5' state changed to StagingInput. [Callback]: ComputePilot '542d6ea8f8cdba3e87873fd3' state changed to PendingActive. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd7' state changed to StagingInput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd9' state changed to PendingExecution. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fda' state changed to StagingInput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdc' state changed to StagingInput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdb' state changed to PendingExecution. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd8' state changed to StagingInput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd5' state changed to PendingExecution. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd7' state changed to PendingExecution. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fda' state changed to PendingExecution. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdc' state changed to PendingExecution. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd8' state changed to PendingExecution. [Callback]: ComputePilot '542d6ea8f8cdba3e87873fd3' state changed to Active. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd6' state changed to Scheduling. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd6' state changed to Executing. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd9' state changed to Scheduling. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd9' state changed to Executing. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdb' state changed to Scheduling. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdb' state changed to Executing. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd5' state changed to Scheduling. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd5' state changed to Executing. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd7' state changed to Scheduling. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd7' state changed to Executing. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fda' state changed to Scheduling. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fda' state changed to Executing. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdc' state changed to Scheduling. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdc' state changed to Executing. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd8' state changed to Scheduling. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd8' state changed to Executing. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd8' state changed to StagingOutput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdb' state changed to StagingOutput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd5' state changed to StagingOutput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd7' state changed to StagingOutput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fda' state changed to StagingOutput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdc' state changed to StagingOutput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd6' state changed to StagingOutput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd9' state changed to StagingOutput. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdb' state changed to Done. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd5' state changed to Done. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd8' state changed to Done. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fda' state changed to Done. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fdc' state changed to Done. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd9' state changed to Done. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd6' state changed to Done. [Callback]: ComputeUnit '542d6eaff8cdba3e87873fd7' state changed to Done. Total Simulation Time : 122.460048914 Simulation Execution Time : Starting Analysis Submitting COCO Compute Unit [Callback]: ComputeUnit '542d6f29f8cdba3e87873fdd' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6f29f8cdba3e87873fdd' state changed to StagingInput. [Callback]: ComputeUnit '542d6f29f8cdba3e87873fdd' state changed to PendingExecution. [Callback]: ComputeUnit '542d6f29f8cdba3e87873fdd' state changed to Executing. [Callback]: ComputeUnit '542d6f29f8cdba3e87873fdd' state changed to StagingOutput. [Callback]: ComputeUnit '542d6f29f8cdba3e87873fdd' state changed to Failed. Log: Output transfer failed: File does not exist: '/home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/unit-542d6f29f8cdba3e87873fdd/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/output_file_transfer_worker.py", line 148, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '/home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/unit-542d6f29f8cdba3e87873fdd/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

Analysis Execution time : Cycle 1 Starting Simulation [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe0' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fde' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fdf' state changed to StagingInput. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe0' state changed to StagingInput. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe1' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe2' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe3' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe4' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe5' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fde' state changed to StagingInput. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fdf' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min11.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min11.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d6f54f8cdba3e87873fe1' state changed to StagingInput. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe0' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min12.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min12.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d6f54f8cdba3e87873fe2' state changed to StagingInput. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fde' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d6f54f8cdba3e87873fe3' state changed to StagingInput. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe1' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min13.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min13.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d6f54f8cdba3e87873fe4' state changed to StagingInput. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe2' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min14.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min14.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d6f54f8cdba3e87873fe3' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min15.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min15.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d6f54f8cdba3e87873fe5' state changed to StagingInput. [Callback]: ComputeUnit '542d6f54f8cdba3e87873fe4' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min16.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min16.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d6f54f8cdba3e87873fe5' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min17.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min17.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

Total Simulation Time : 14.4104521275 Simulation Execution Time : Starting Analysis Submitting COCO Compute Unit [Callback]: ComputeUnit '542d6f62f8cdba3e87873fe6' state changed to StagingInput. [Callback]: ComputeUnit '542d6f62f8cdba3e87873fe6' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

Analysis Execution time : [Callback]: ComputePilot '542d6ea8f8cdba3e87873fd3' state changed to Canceled. Exception in thread OutputFileTransferWorker-1 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in bootstrap_inner File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/output_file_transfer_worker.py", line 79, in run File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/collection.py", line 1677, in find_and_modify File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 445, in command File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 345, in _command File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1058, in next File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1002, in _refresh File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 915, in send_message File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/mongo_client.py", line 1207, in _send_message_with_response <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error' Exception in thread InputFileTransferWorker-1 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in bootstrap_inner File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 232, in run File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/collection.py", line 1677, in find_and_modify File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 445, in command File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 345, in _command File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1058, in next File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1002, in _refresh File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 915, in send_message File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/mongo_client.py", line 1207, in _send_message_with_response <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

vivek-bala commented 9 years ago

Seems like to coco compute unit failed. Could you paste the contents of STDOUT, STDERR and a shell script which is probably named radicalpilot*.sh inside the compute unit folder unit-542d6f29f8cdba3e87873fdd on Stampede.

ashkurti commented 9 years ago

login3.stampede(73)$ cd pilot-542d6ea8f8cdba3e87873fd3 login3.stampede(74)$ ls AGENT.LOG unit-542d6eaff8cdba3e87873fd7 unit-542d6f54f8cdba3e87873fde unit-542d6f54f8cdba3e87873fe5 AGENT.STDERR unit-542d6eaff8cdba3e87873fd8 unit-542d6f54f8cdba3e87873fdf unit-542d6f62f8cdba3e87873fe6 AGENT.STDOUT unit-542d6eaff8cdba3e87873fd9 unit-542d6f54f8cdba3e87873fe0 virtualenv defaultbootstrapper.sh unit-542d6eaff8cdba3e87873fda unit-542d6f54f8cdba3e87873fe1 virtualenv-1.9 radical-pilot-agent.py unit-542d6eaff8cdba3e87873fdb unit-542d6f54f8cdba3e87873fe2 virtualenv-1.9.tar.gz unit-542d6eaff8cdba3e87873fd5 unit-542d6eaff8cdba3e87873fdc unit-542d6f54f8cdba3e87873fe3 unit-542d6eaff8cdba3e87873fd6 unit-542d6f29f8cdba3e87873fdd unit-542d6f54f8cdba3e87873fe4 login3.stampede(75)$ cat AGENT.STDERR % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1957k 100 1957k 0 0 1696k 0 0:00:01 0:00:01 --:--:-- 2899k warning: no files found matching '.html' under directory 'docs' warning: no previously-included files matching '_.txt' found under directory 'docs/_build' no previously-included directories found matching 'docs/_build/_sources' no previously-included directories found matching 'libcloud/test/secrets.py'

login3.stampede(79)$ cat AGENT.STDOUT ################################################################################

Bootstrapper running on host: c545-004.stampede.tacc.utexas.edu.

################################################################################

Environment of bootstrapper process:

MKLROOT=/opt/apps/intel/13/composer_xe_2013.2.146/mkl SLURM_NODELIST=c545-004,c547-[301,904],c548-802 SLURM_CHECKPOINT_IMAGE_DIR=/home1/02998/ardi MANPATH=/opt/apps/intel13/mvapich2/1.9/man:/opt/apps/intel/13/composer_xe_2013.2.146/man/en_US:/opt/apps/intel/13/composer_xe_2013.2.146/man/en_US:/opt/apps/intel/13/vtune_amplifier_xe_2013/man:/usr/local/man:/usr/share/man:/usr/X11R6/man:/usr/kerberos/man:/usr/man:/usr/local/man SLURM_JOB_NAME=SAGAPythonSLURMJob HOSTNAME=c545-004.stampede.tacc.utexas.edu TACC_XALT_DIR=/opt/apps/xalt/0.4.4/ ModuleTable003=VD17fSxbInNob3J0Il09ImNsdXN0ZXIiLFsic3RhdHVzIl09ImFjdGl2ZSIsfSxbImNsdXN0ZXItcGF0aHMiXT17WyJGTiJdPSIvb3B0L21vZHVsZWZpbGVzL2NsdXN0ZXItcGF0aHMiLFsiZGVmYXVsdCJdPTAsWyJmdWxsTmFtZSJdPSJjbHVzdGVyLXBhdGhzIixbImxvYWRPcmRlciJdPTMscHJvcFQ9e30sWyJzaG9ydCJdPSJjbHVzdGVyLXBhdGhzIixbInN0YXR1cyJdPSJhY3RpdmUiLH0saW50ZWw9e1siRk4iXT0iL29wdC9hcHBzL21vZHVsZWZpbGVzL2ludGVsLzEzLjAuMi4xNDYiLFsiZGVmYXVsdCJdPTEsWyJmdWxsTmFtZSJdPSJpbnRlbC8xMy4wLjIuMTQ2IixbImxvYWRPcmRlciJdPTQscHJvcFQ9e30sWyJzaG9ydCJdPSJpbnRlbCIsWyJzdGF0dXMiXT0iYWN0aXZl SLURM_TOPOLOGY_ADDR=c545-004 SLURMD_NODENAME=c545-004 SLURM_PRIO_PROCESS=0 IPPROOT=/opt/apps/intel/13/composer_xe_2013.2.146/ipp MPICH_HOME=/opt/apps/intel13/mvapich2/1.9 SLURM_NODE_ALIASES=(null) SHELL=/bin/bash TERM=vt100 IIDB_BIN=/opt/apps/intel/13/composer_xe_2013.2.146/bin/intel64 HISTSIZE=1000 SSH_CLIENT=128.243.118.196 42740 22 LMOD_SYSTEM_DEFAULT_MODULES=TACC LMOD_DEFAULT_MODULEPATH=/opt/apps/xsede/modulefiles:/opt/apps/modulefiles:/opt/modulefiles SLURM_TOPOLOGY_ADDR_PATTERN=node TMPDIR=/tmp MIC_I_MPI_OFA_ADAPTER_NAME=mlx4_0 LMOD_PKG=/opt/apps/lmod/lmod QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/154 MIC_LD_LIBRARY_PATH=/opt/apps/intel/13/composer_xe_2013.2.146/tbb/lib/mic:/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/mic:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/mic/myo/lib:/opt/apps/intel/13/composer_xe_2013.2.146/tbb/lib/mic:/opt/apps/intel/13/composer_xe_2013.2.146/mkl/lib/mic SLURM_TACC_RUNLIMIT_MINS=60 LC_ALL=en_US.UTF-8 USER=ardi ARMCI_OPENIB_DEVICE=mlx4_0 SLURM_NNODES=4 LSCOLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arj=01;31:.taz=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lz=01;31:.xz=01;31:.bz2=01;31:.tbz=01;31:.tbz2=01;31:.bz=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.rar=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:_.xspf=01;36: LD_LIBRARY_PATH=/opt/apps/intel13/mvapich2/1.9/lib:/opt/apps/intel13/mvapich2/1.9/lib/shared:/opt/apps/intel/13/composer_xe_2013.2.146/tbb/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/apps/intel/13/composer_xe_2013.2.146/mpirt/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/ipp/../compiler/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/ipp/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/mkl/lib/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/tbb/lib/intel64 TACC_XALT_BIN=/opt/apps/xalt/0.4.4/bin SLURM_TACC_NODES=1 IFC_BIN=/opt/apps/intel/13/composer_xe_2013.2.146/bin/intel64 TACC_MKL_LIB=/opt/apps/intel/13/composer_xe_2013.2.146/mkl/lib/intel64 ModuleTable004=Iix9LG12YXBpY2gyPXtbIkZOIl09Ii9vcHQvYXBwcy9pbnRlbDEzL21vZHVsZWZpbGVzL212YXBpY2gyLzEuOWEyIixbImRlZmF1bHQiXT0xLFsiZnVsbE5hbWUiXT0ibXZhcGljaDIvMS45YTIiLFsibG9hZE9yZGVyIl09NSxwcm9wVD17fSxbInNob3J0Il09Im12YXBpY2gyIixbInN0YXR1cyJdPSJhY3RpdmUiLH0seGFsdD17WyJGTiJdPSIvb3B0L2FwcHMvbW9kdWxlZmlsZXMveGFsdC8wLjQuNC5sdWEiLFsiZGVmYXVsdCJdPTEsWyJmdWxsTmFtZSJdPSJ4YWx0LzAuNC40IixbImxvYWRPcmRlciJdPTYscHJvcFQ9e30sWyJzaG9ydCJdPSJ4YWx0IixbInN0YXR1cyJdPSJhY3RpdmUiLH0sfSxtcGF0aEE9eyIvb3B0L2FwcHMvaW50ZWwxMy9tdmFwaWNoMl8xXzkvbW9kdWxl SLURM_JOBID=4208657 TACC_PUBLIC_MACHINE=1 APPS=/opt/apps SLURM_NTASKS=64 SLURM_TACC_JOBNAME=SAGAPythonSLURMJob LMOD_PREPEND_BLOCK=normal SYSTEM=linux ARCHIVER=ranch.tacc.utexas.edu PATH=/opt/apps/xalt/0.4.4/bin:/opt/apps/intel13/mvapich2/1.9/bin:/opt/apps/intel/13/vtune_amplifier_xe_2013/bin64:/opt/apps/intel/13/composer_xe_2013.2.146/mpirt/bin/intel64:/opt/intel/sep/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/opt/ofed/bin:/opt/ofed/sbin:.:/opt/apps/intel/13/composer_xe_2013.2.146/bin/intel64:/opt/apps/intel/13/composer_xe_2013.2.146/bin/intel64_mic:/opt/apps/intel/13/composer_xe_2013.2.146/debugger/gui/intel64 MAIL=/var/spool/mail/ardi I_MPI_OFA_ADAPTER_NAME=mlx4_0 ICC_BIN=/opt/apps/intel/13/composer_xe_2013.2.146/bin/intel64 TACC_SYSTEM=stampede ModuleTable001=X01vZHVsZVRhYmxlXz17WyJhY3RpdmVTaXplIl09OCxiYXNlTXBhdGhBPXsiL29wdC9hcHBzL3hzZWRlL21vZHVsZWZpbGVzIiwiL29wdC9hcHBzL21vZHVsZWZpbGVzIiwiL29wdC9tb2R1bGVmaWxlcyIsfSxbImNfcmVidWlsZFRpbWUiXT1mYWxzZSxbImNfc2hvcnRUaW1lIl09ZmFsc2UsZmFtaWx5PXtbIk1QSSJdPSJtdmFwaWNoMiIsWyJjb21waWxlciJdPSJpbnRlbCIsfSxpbmFjdGl2ZT17fSxtVD17TGludXg9e1siRk4iXT0iL29wdC9tb2R1bGVmaWxlcy9MaW51eCIsWyJkZWZhdWx0Il09MCxbImZ1bGxOYW1lIl09IkxpbnV4IixbImxvYWRPcmRlciJdPTIscHJvcFQ9e30sWyJzaG9ydCJdPSJMaW51eCIsWyJzdGF0dXMiXT0iYWN0aXZlIix9LFRBQ0M9e1siRk4iXT0i SLURM_TASKS_PER_NODE=16(x4) STOCKYARD=/work/02998/ardi WORK=/work/02998/ardi SLURM_CPUS_PER_TASK=1 SLURM_JOB_ID=4208657 LMOD_SETTARG_CMD=: TBBROOT=/opt/apps/intel/13/composer_xe_2013.2.146/tbb PWD=/home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3 INPUTRC=/etc/inputrc SLURM_QUEUE=normal SLURM_TACC_NCORES_SET=1 XMODIFIERS=@im=none I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1u LMFILES=/opt/modulefiles/TACC-paths:/opt/modulefiles/Linux:/opt/modulefiles/cluster-paths:/opt/apps/modulefiles/intel/13.0.2.146:/opt/apps/intel13/modulefiles/mvapich2/1.9a2:/opt/apps/modulefiles/xalt/0.4.4.lua:/opt/modulefiles/cluster:/opt/modulefiles/TACC LANG=en_US.UTF-8 MODULEPATH=/opt/apps/intel13/mvapich2_1_9/modulefiles:/opt/apps/intel13/modulefiles:/opt/apps/xsede/modulefiles:/opt/apps/modulefiles:/opt/modulefiles LOADEDMODULES=TACC-paths:Linux:cluster-paths:intel/13.0.2.146:mvapich2/1.9a2:xalt/0.4.4:cluster:TACC _ModuleTableSz=5 SLURM_TACC_ACCOUNT=TG-MCB090174 SLURM_NODEID=0 MIC_ENV_PREFIX=MIC SLURM_SUBMIT_DIR=/home1/02998/ardi LMOD_CMD=/opt/apps/lmod/lmod/libexec/lmod ModuleTable005=ZmlsZXMiLCIvb3B0L2FwcHMvaW50ZWwxMy9tb2R1bGVmaWxlcyIsIi9vcHQvYXBwcy94c2VkZS9tb2R1bGVmaWxlcyIsIi9vcHQvYXBwcy9tb2R1bGVmaWxlcyIsIi9vcHQvbW9kdWxlZmlsZXMiLH0sWyJzeXN0ZW1CYXNlTVBBVEgiXT0iL29wdC9hcHBzL3hzZWRlL21vZHVsZWZpbGVzOi9vcHQvYXBwcy9tb2R1bGVmaWxlczovb3B0L21vZHVsZWZpbGVzIixbInZlcnNpb24iXT0yLH0= SLURM_NPROCS=64 SLURM_TASK_PID=42427 SLURM_CPUS_ON_NODE=16 TACC_MKL_INC=/opt/apps/intel/13/composer_xe_2013.2.146/mkl/include ENVIRONMENT=BATCH SLURM_PROCID=0 SLURM_JOB_NODELIST=c545-004,c547-[301,904],c548-802 HOME=/home1/02998/ardi SHLVL=3 TACC_DOMAIN=stampede TACC_FAMILY_COMPILER=intel SLURM_LOCALID=0 IFC_LIB=/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64 ModuleTable002=L29wdC9tb2R1bGVmaWxlcy9UQUNDIixbImRlZmF1bHQiXT0wLFsiZnVsbE5hbWUiXT0iVEFDQyIsWyJsb2FkT3JkZXIiXT04LHByb3BUPXt9LFsic2hvcnQiXT0iVEFDQyIsWyJzdGF0dXMiXT0iYWN0aXZlIix9LFsiVEFDQy1wYXRocyJdPXtbIkZOIl09Ii9vcHQvbW9kdWxlZmlsZXMvVEFDQy1wYXRocyIsWyJkZWZhdWx0Il09MCxbImZ1bGxOYW1lIl09IlRBQ0MtcGF0aHMiLFsibG9hZE9yZGVyIl09MSxwcm9wVD17fSxbInNob3J0Il09IlRBQ0MtcGF0aHMiLFsic3RhdHVzIl09ImFjdGl2ZSIsfSxjbHVzdGVyPXtbIkZOIl09Ii9vcHQvbW9kdWxlZmlsZXMvY2x1c3RlciIsWyJkZWZhdWx0Il09MCxbImZ1bGxOYW1lIl09ImNsdXN0ZXIiLFsibG9hZE9yZGVyIl09Nyxwcm9w SLURM_JOB_CPUS_PER_NODE=16(x4) SLURM_SUBMIT_HOST=login4.stampede.tacc.utexas.edu SLURM_GTIDS=0 BASH_ENV=/etc/tacc/tacc_functions LOGNAME=ardi MV2_IBA_HCA=mlx4_0 ICC_LIB=/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/intel64 TACC_FAMILY_MPI=mvapich2 CVS_RSH=ssh QTLIB=/usr/lib64/qt-3.3/lib SSH_CONNECTION=128.243.118.196 42740 129.114.62.14 22 MODULESHOME=/opt/apps/lmod/lmod SINK_LD_LIBRARY_PATH=/opt/apps/intel/13/composer_xe_2013.2.146/compiler/lib/mic:/opt/apps/intel/13/composer_xe_2013.2.146/mkl/lib/mic SLURM_JOB_NUM_NODES=4 LESSOPEN=|/usr/bin/lesspipe.sh %s OMP_NUM_THREADS=1 PKG_CONFIG_PATH=/opt/apps/intel13/mvapich2/1.9/lib/pkgconfig __Init_Default_Modules=1 LMOD_FULL_SETTARG_SUPPORT=no INFOPATH=/opt/apps/intel13/mvapich2/1.9/doc LMOD_FAMILY_COMPILER=intel DISPLAY=localhost:70.0 ARCHIVE=/home/02998/ardi LMOD_DIR=/opt/apps/lmod/lmod/libexec INCLUDE=/opt/apps/intel/13/composer_xe_2013.2.146/ipp/include:/opt/apps/intel/13/composer_xe_2013.2.146/mkl/include G_BROKEN_FILENAMES=1 LMOD_COLORIZE=yes SCRATCH=/scratch/02998/ardi MIC_I_MPI_DAPL_PROVIDER=ofa-v2-mlx4_0-1u LMOD_FAMILY_MPI=mvapich2 TACC_MKL_DIR=/opt/apps/intel/13/composer_xe_2013.2.146/mkl TACC_MPI_GETMODE=mvapich2_ssh SLURM_TACC_NNODES_SET=0 SLURM_TACC_CORES=64 BASH_FUNC_module%%=() { eval $LMOD_CMD bash $*; [ $? = 0 ] && eval $(${LMOD_SETTARGCMD:-:} -s sh) } =/usr/bin/printenv

################################################################################

Running pre-bootstrapping command

CMDLINE: module purge

################################################################################

Running pre-bootstrapping command

CMDLINE: module load TACC

################################################################################

Running pre-bootstrapping command

CMDLINE: module load cluster

################################################################################

Running pre-bootstrapping command

CMDLINE: module load Linux

################################################################################

Running pre-bootstrapping command

CMDLINE: module load mvapich2

################################################################################

Running pre-bootstrapping command

CMDLINE: module load python/2.7.3-epd-7.3.2

################################################################################

Running pre-bootstrapping command

CMDLINE: module unload xalt

################################################################################

Downloading and installing virtualenv

CMDLINE: curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.tar.gz

virtualenv-1.9/ virtualenv-1.9/AUTHORS.txt virtualenv-1.9/bin/ virtualenv-1.9/bin/rebuild-script.py virtualenv-1.9/bin/refresh-support-files.py virtualenv-1.9/docs/ virtualenv-1.9/docs/_theme/ virtualenv-1.9/docs/_theme/nature/ virtualenv-1.9/docs/_theme/nature/static/ virtualenv-1.9/docs/_theme/nature/static/nature.css_t virtualenv-1.9/docs/_theme/nature/static/pygments.css virtualenv-1.9/docs/_theme/nature/theme.conf virtualenv-1.9/docs/conf.py virtualenv-1.9/docs/index.txt virtualenv-1.9/docs/make.bat virtualenv-1.9/docs/Makefile virtualenv-1.9/docs/news.txt virtualenv-1.9/LICENSE.txt virtualenv-1.9/MANIFEST.in virtualenv-1.9/PKG-INFO virtualenv-1.9/README.rst virtualenv-1.9/scripts/ virtualenv-1.9/scripts/virtualenv virtualenv-1.9/setup.cfg virtualenv-1.9/setup.py virtualenv-1.9/virtualenv.egg-info/ virtualenv-1.9/virtualenv.egg-info/dependency_links.txt virtualenv-1.9/virtualenv.egg-info/entry_points.txt virtualenv-1.9/virtualenv.egg-info/not-zip-safe virtualenv-1.9/virtualenv.egg-info/PKG-INFO virtualenv-1.9/virtualenv.egg-info/SOURCES.txt virtualenv-1.9/virtualenv.egg-info/top_level.txt virtualenv-1.9/virtualenv.py virtualenv-1.9/virtualenv_embedded/ virtualenv-1.9/virtualenv_embedded/activate.bat virtualenv-1.9/virtualenv_embedded/activate.csh virtualenv-1.9/virtualenv_embedded/activate.fish virtualenv-1.9/virtualenv_embedded/activate.ps1 virtualenv-1.9/virtualenv_embedded/activate.sh virtualenv-1.9/virtualenv_embedded/activate_this.py virtualenv-1.9/virtualenv_embedded/deactivate.bat virtualenv-1.9/virtualenv_embedded/distribute_from_egg.py virtualenv-1.9/virtualenv_embedded/distribute_setup.py virtualenv-1.9/virtualenv_embedded/distutils-init.py virtualenv-1.9/virtualenv_embedded/distutils.cfg virtualenv-1.9/virtualenv_embedded/ez_setup.py virtualenv-1.9/virtualenv_embedded/site.py virtualenv-1.9/virtualenv_support/ virtualenv-1.9/virtualenv_support/init.py virtualenv-1.9/virtualenv_support/distribute-0.6.34.tar.gz virtualenv-1.9/virtualenv_support/pip-1.3.tar.gz virtualenv-1.9/virtualenv_support/setuptools-0.6c11-py2.5.egg virtualenv-1.9/virtualenv_support/setuptools-0.6c11-py2.6.egg virtualenv-1.9/virtualenv_support/setuptools-0.6c11-py2.7.egg

################################################################################

Creating virtualenv

CMDLINE: /opt/apps/python/epd/7.3.2/bin/python virtualenv-1.9/virtualenv.py /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/

PYTHONHOME is set. You must activate the virtualenv before using it New python executable in /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin/python Installing setuptools............done. Installing pip...............done.

################################################################################

Downgrading pip to 1.2.1

CMDLINE: easy_install pip==1.2.1

Searching for pip==1.2.1 Reading http://pypi.python.org/simple/pip/ Best match: pip 1.2.1 Downloading https://pypi.python.org/packages/source/p/pip/pip-1.2.1.tar.gz#md5=db8a6d8a4564d3dc7f337ebed67b1a85 Processing pip-1.2.1.tar.gz Running pip-1.2.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-iAmVno/pip-1.2.1/egg-dist-tmp-yRnHgy Removing pip 1.3 from easy-install.pth file Adding pip 1.2.1 to easy-install.pth file Installing pip script to /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin Installing pip-2.7 script to /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin

Installed /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg Processing dependencies for pip==1.2.1 Finished processing dependencies for pip==1.2.1

################################################################################

install/upgrade Apache-LibCloud

CMDLINE: easy_install --upgrade apache-libcloud

Searching for apache-libcloud Reading http://pypi.python.org/simple/apache-libcloud/ Best match: apache-libcloud 0.15.1 Downloading https://pypi.python.org/packages/source/a/apache-libcloud/apache-libcloud-0.15.1.zip#md5=b65fa11b7af5406ee50c25fb6af25975 Processing apache-libcloud-0.15.1.zip Running apache-libcloud-0.15.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-uBCds0/apache-libcloud-0.15.1/egg-dist-tmp-P2aZ0V Adding apache-libcloud 0.15.1 to easy-install.pth file

Installed /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/lib/python2.7/site-packages/apache_libcloud-0.15.1-py2.7.egg Processing dependencies for apache-libcloud Finished processing dependencies for apache-libcloud

################################################################################

install/upgrade SAGA-Python

CMDLINE: pip install --upgrade saga-python

Downloading/unpacking saga-python Running setup.py egg_info for package saga-python

Requirement already up-to-date: apache-libcloud in ./virtualenv/lib/python2.7/site-packages/apache_libcloud-0.15.1-py2.7.egg (from saga-python) Downloading/unpacking radical.utils (from saga-python) Running setup.py egg_info for package radical.utils version: 0.7.7 (v0.7.7)

Downloading/unpacking colorama (from radical.utils->saga-python) Downloading colorama-0.3.2.tar.gz Running setup.py egg_info for package colorama

Downloading/unpacking pymongo (from radical.utils->saga-python) Running setup.py egg_info for package pymongo

Downloading/unpacking nose (from radical.utils->saga-python) Running setup.py egg_info for package nose

no previously-included directories found matching 'doc/.build'

Installing collected packages: saga-python, radical.utils, colorama, pymongo, nose Running setup.py install for saga-python changing mode of build/scripts-2.7/sagapython-version from 600 to 755

changing mode of /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin/sagapython-version to 755

Running setup.py install for radical.utils version: 0.7.7 (v0.7.7) changing mode of build/scripts-2.7/dump_mongodb.py from 600 to 755 Skipping installation of /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/lib/python2.7/site-packages/radical/init.py (namespace package)

Installing /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/lib/python2.7/site-packages/radical.utils-0.7.7-py2.7-nspkg.pth
changing mode of /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin/dump_mongodb.py to 755

Running setup.py install for colorama

Running setup.py install for pymongo building 'bson._cbson' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 -fPIC -Ibson -I/opt/apps/python/epd/7.3.2/include/python2.7 -c bson/_cbsonmodule.c -o build/temp.linux-x86_64-2.7/bson/_cbsonmodule.o gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 -fPIC -Ibson -I/opt/apps/python/epd/7.3.2/include/python2.7 -c bson/time64.c -o build/temp.linux-x86_64-2.7/bson/time64.o gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 -fPIC -Ibson -I/opt/apps/python/epd/7.3.2/include/python2.7 -c bson/buffer.c -o build/temp.linux-x86_64-2.7/bson/buffer.o gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 -fPIC -Ibson -I/opt/apps/python/epd/7.3.2/include/python2.7 -c bson/encoding_helpers.c -o build/temp.linux-x86_64-2.7/bson/encoding_helpers.o gcc -pthread -shared -g build/temp.linux-x86_64-2.7/bson/_cbsonmodule.o build/temp.linux-x86_64-2.7/bson/time64.o build/temp.linux-x86_64-2.7/bson/buffer.o build/temp.linux-x86_64-2.7/bson/encoding_helpers.o -L/opt/apps/python/epd/7.3.2/lib -lpython2.7 -o build/lib.linux-x86_64-2.7/bson/_cbson.so building 'pymongo._cmessage' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 -fPIC -Ibson -I/opt/apps/python/epd/7.3.2/include/python2.7 -c pymongo/_cmessagemodule.c -o build/temp.linux-x86_64-2.7/pymongo/_cmessagemodule.o gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -O2 -fPIC -Ibson -I/opt/apps/python/epd/7.3.2/include/python2.7 -c bson/buffer.c -o build/temp.linux-x86_64-2.7/bson/buffer.o gcc -pthread -shared -g build/temp.linux-x86_64-2.7/pymongo/_cmessagemodule.o build/temp.linux-x86_64-2.7/bson/buffer.o -L/opt/apps/python/epd/7.3.2/lib -lpython2.7 -o build/lib.linux-x86_64-2.7/pymongo/_cmessage.so

Running setup.py install for nose

no previously-included directories found matching 'doc/.build'
Installing nosetests script to /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin
Installing nosetests-2.7 script to /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin

Successfully installed saga-python radical.utils colorama pymongo nose Cleaning up...

################################################################################

install/upgrade python-hostlist

CMDLINE: pip install --upgrade python-hostlist

Downloading/unpacking python-hostlist Downloading python-hostlist-1.14.tar.gz Running setup.py egg_info for package python-hostlist

Installing collected packages: python-hostlist Running setup.py install for python-hostlist changing mode of build/scripts-2.7/hostlist from 600 to 755 changing mode of build/scripts-2.7/hostgrep from 600 to 755 changing mode of build/scripts-2.7/pshbak from 600 to 755 changing mode of build/scripts-2.7/dbuck from 600 to 755

changing mode of /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin/hostlist to 755
changing mode of /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin/pshbak to 755
changing mode of /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin/dbuck to 755
changing mode of /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/virtualenv/bin/hostgrep to 755

Successfully installed python-hostlist Cleaning up...

################################################################################

install/upgrade pymongo

CMDLINE: pip install --upgrade pymongo

Requirement already up-to-date: pymongo in ./virtualenv/lib/python2.7/site-packages Cleaning up...

################################################################################

Launching radical-pilot-agent for 64 cores.

CMDLINE: python radical-pilot-agent.py -b 0 -c 64 -d 50 -j SSH -k IBRUN -l SLURM -m ec2-184-72-89-141.compute-1.amazonaws.com:27017 -n radicalpilot -p 542d6ea8f8cdba3e87873fd3 -s 542d6ea5f8cdba3e87873fd1 -t 60 -v 0.20

CLEANUP:

login3.stampede(6)$ cat default_bootstrapper.sh

!/bin/bash -l

-----------------------------------------------------------------------------

Copyright 2013-2014, radical@rutgers.edu

License under the MIT License

#

This script launches a radical.pilot compute pilot.

#

-----------------------------------------------------------------------------

global variables

# CLEANUP= CORES= DBNAME= DBURL= DEBUG= VIRTENV= GLOBAL_VIRTENV= LRMS= MPI_LAUNCH_METHOD= PREBOOTSTRAP= PILOTID= PYTHON= RUNTIME= SESSIONID= TASK_LAUNCH_METHOD= VERSION= SANDBOX=pwd

-----------------------------------------------------------------------------

contains(string, substring)

#

Returns 0 if the specified string contains the specified substring,

otherwise returns 1.

contains() { string="$1" substring="$2" if test "${string#*$substring}" != "$string" then return 0 # $substring is in $string else return 1 # $substring is not in $string fi }

-----------------------------------------------------------------------------

print out script usage help

# usage() { cat << EOF > /dev/stderr usage: $0 options

This script launches a RADICAL-Pilot agent.

OPTIONS: -a The name of project / allocation to charge.

-b enable agent benchmarking

-c Number of requested cores.

-d Specify debug level.

-e List of commands to run before bootstrapping.

-f Tunnel endpoint for connection forwarding.

-g Global shared virtualenv (create if missing)

-h Show this message.

-i The Python interpreter to use, e.g., python2.6. (default is '/usr/bin/python')

-j Task launch method.

-k MPI launch method.

-l Type of Local Resource Management System.

-m Address and port of the coordination service host (MongoDB).

-n The name of the database.

-p The unique identifier (uid) of the pilot.

-s The unique identifier (uid) of the session.

-t Runtime in minutes.

-u sandbox is user defined

-v Version - the RADICAL-Pilot package version.

-w The working directory (sandbox) of the pilot. (default is '.')

-x Cleanup - delete pilot sandbox, virtualenv etc. after completion

EOF }

-----------------------------------------------------------------------------

bootstrap virtualenv - we always use the latest version from GitHub

# installvenv() {

first argument is the virtenv target

VIRTENV=$1
# create a fresh virtualenv. we use an older 1.9.x version of 
# virtualenv as this seems to work more reliable than newer versions.
# If we can't download, we try to move on with the system virtualenv.
CURL_CMD="curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.9.tar.gz"
echo ""
echo "################################################################################"
echo "## Downloading and installing virtualenv"
echo "## CMDLINE: $CURL_CMD"
$CURL_CMD
OUT=$?
if [[ $OUT != 0 ]]; then
    echo "WARNING: Couldn't download virtualenv via curl! Using system version."
    BOOTSTRAP_CMD="virtualenv $VIRTENV"
else :
    tar xvfz virtualenv-1.9.tar.gz
    OUT=$?
    if [[ $OUT != 0 ]]; then
       echo "Couldn't unpack virtualenv! ABORTING"
       exit 1
    fi

    BOOTSTRAP_CMD="$PYTHON virtualenv-1.9/virtualenv.py $VIRTENV"
fi

echo ""
echo "################################################################################"
echo "## Creating virtualenv"
echo "## CMDLINE: $BOOTSTRAP_CMD"
$BOOTSTRAP_CMD
OUT=$?
if [[ $OUT != 0 ]]; then
    echo "Couldn't bootstrap virtualenv! ABORTING"
    exit 1
fi

# activate the virtualenv
source $VIRTENV/bin/activate

DOWNGRADE_PIP_CMD="easy_install pip==1.2.1"
echo ""
echo "################################################################################"
echo "## Downgrading pip to 1.2.1"
echo "## CMDLINE: $DOWNGRADE_PIP_CMD"
$DOWNGRADE_PIP_CMD
OUT=$?
if [[ $OUT != 0 ]]; then
    echo "Couldn't downgrade pip! Using default version (if it exists)"
fi

#UPDATE_SETUPTOOLS_CMD="pip install --upgrade setuptools"
#echo ""
#echo "################################################################################"
#echo "## Updating virtualenv"
#echo "## CMDLINE: $UPDATE_SETUPTOOLS_CMD"
#$UPDATE_SETUPTOOLS_CMD
#OUT=$?
#if [ $OUT -ne 0 ]; then
#    echo "Couldn't update virtualenv! ABORTING"
#    exit 1
#fi

# On india/fg 'pip install saga-python' does not work as pip fails to
# install apache-libcloud (missing bz2 compression).  We thus install that
# dependency via easy_install.
EI_CMD="easy_install --upgrade apache-libcloud"
echo ""
echo "################################################################################"
echo "## install/upgrade Apache-LibCloud"
echo "## CMDLINE: $EI_CMD"
$EI_CMD
OUT=$?
if [ $OUT -ne 0 ];then
    echo "Couldn't install/upgrade apache-libcloud! Lets see how far we get ..."
fi

# Now pip install should work...
PIP_CMD="pip install --upgrade saga-python"
EA_CMD="easy_install --upgrade saga-python"
echo ""
echo "################################################################################"
echo "## install/upgrade SAGA-Python"
echo "## CMDLINE: $PIP_CMD"
$PIP_CMD
OUT=$?
if [ $OUT -ne 0 ];then
    echo "pip install failed, trying easy_install ..."
    $EI_CMD
    OUT=$?
    if [ $OUT -ne 0 ];then
        echo "Couldn't install/upgrade SAGA-Python! Lets see how far we get ..."
    fi
fi

PIP_CMD="pip install --upgrade python-hostlist"
EI_CMD="easy_install --upgrade python-hostlist"
echo ""
echo "################################################################################"
echo "## install/upgrade python-hostlist"
echo "## CMDLINE: $PIP_CMD"
$PIP_CMD
OUT=$?
if [ $OUT -ne 0 ];then
    echo "pip install failed, trying easy_install ..."
    $EI_CMD
    OUT=$?
    if [ $OUT -ne 0 ];then
        echo "Easy install failed too, couldn't install python-hostlist!  Lets see how far we get..."
    fi
fi

# pymongo should be pulled by saga, via utils.  But whatever...
PIP_CMD="pip install --upgrade pymongo"
EI_CMD="easy_install --upgrade pymongo"
echo ""
echo "################################################################################"
echo "## install/upgrade pymongo"
echo "## CMDLINE: $PIP_CMD"
$PIP_CMD
OUT=$?
if [ $OUT -ne 0 ];then
    echo "pip install failed, trying easy_install ..."
    $EI_CMD
    OUT=$?
    if [ $OUT -ne 0 ];then
        echo "Easy install failed too, couldn't install pymongo! Oh well..."
    fi
fi

}

-----------------------------------------------------------------------------

Find available port on the remote host where we can bind to

# find_available_port() { RANGE="23000..23100" echo "" echo "################################################################################" echo "## Searching for available TCP port for tunnel in range $RANGE." host=$1 for port in $(eval echo {$RANGE}); do

    # Try to make connection
    (bash -c "(>/dev/tcp/$host/$port)" 2>/dev/null) &
    # Wait for 1 second
    read -t1
    # Kill child
    kill $! 2>/dev/null
    # If the kill command succeeds, assume that we have found our match!
    if [ "$?" == "0" ]; then
        break
    fi

    # Reset port, so that the last port doesn't get chosen in error
    port=
done

# Wait for children
wait 2>/dev/null

# Assume the most recent port is available
AVAILABLE_PORT=$port

}

-----------------------------------------------------------------------------

MAIN

#

Report where we are, as this is not always what you expect ;-)

echo "################################################################################" echo "## Bootstrapper running on host: hostname -f."

Print environment, useful for debugging

echo "" echo "################################################################################" echo "## Environment of bootstrapper process:" printenv

parse command line arguments

USER_SANDBOX=0 BENCHMARK=0 while getopts "abc:d:e:f:g:hi:j:k:l:m:n:op:qrs:t:uv:w:x:yz" OPTION; do case $OPTION in b)

Passed to agent

        BENCHMARK=1
        ;;
    c)
        # Passed to agent
        CORES=$OPTARG
        ;;
    d)
        # Passed to agent
        DEBUG=$OPTARG
        ;;
    e)
        PREBOOTSTRAP=$OPTARG
        # Note: Executed inline here because -e can be passed multiple times.
        echo ""
        echo "################################################################################"
        echo "## Running pre-bootstrapping command"
        echo "## CMDLINE: $PREBOOTSTRAP"
        $PREBOOTSTRAP
        OUT=$?
        if [[ $OUT -ne 0 ]]; then
            echo "Error running pre-boostrapping command! ABORTING"
            exit 1
        fi
        ;;
    f)
        FORWARD_TUNNEL_ENDPOINT=$OPTARG
        ;;
    g)
        GLOBAL_VIRTENV=$OPTARG
        ;;
    h)
        usage
        exit 1
        ;;
    i)
        PYTHON=$OPTARG
        ;;
    j)
        # Passed to agent
        TASK_LAUNCH_METHOD=$OPTARG
        ;;
    k)
        # Passed to agent
        MPI_LAUNCH_METHOD=$OPTARG
        ;;
    l)
        # Passed to agent
        LRMS=$OPTARG
        ;;
    m)
        # Passed to agent, possibly after rewrite for proxy
        DBURL=$OPTARG
        ;;
    n)
        # Passed to agent
        DBNAME=$OPTARG
        ;;
    p)
        # Passed to agent
        PILOTID=$OPTARG
        ;;
    s)
        # Passed to agent
        SESSIONID=$OPTARG
        ;;
    t)
        # Passed to agent
        RUNTIME=$OPTARG
        ;;
    u)
        USER_SANDBOX=1
        ;;
    v)
        # Passed to agent
        VERSION=$OPTARG
        ;;
    w)
        SANDBOX=$OPTARG
        ;;
    x)
        CLEANUP=$OPTARG
        ;;
    *)
        echo "Unknown option: $OPTION=$OPTARG"
        usage
        exit
        ;;
esac

done

Check that mandatory arguments are set

(Currently all that are passed through to the agent)

if [[ -z $CORES ]] ||\ [[ -z $DEBUG ]] ||\ [[ -z $DBNAME ]] ||\ [[ -z $DBURL ]] ||\ [[ -z $LRMS ]] ||\ [[ -z $MPI_LAUNCH_METHOD ]] ||\ [[ -z $PILOTID ]] ||\ [[ -z $RUNTIME ]] ||\ [[ -z $SESSIONID ]] ||\ [[ -z $TASK_LAUNCH_METHOD ]] ||\ [[ -z $VERSION ]]; then echo "Missing option" usage exit 1 fi

If the host that will run the agent is not capable of communication

with the outside world directly, we will setup a tunnel.

if [[ $FORWARD_TUNNEL_ENDPOINT ]]; then

echo ""
echo "################################################################################"
echo "## Setting up forward tunnel for MongoDB to $FORWARD_TUNNEL_ENDPOINT."

find_available_port $FORWARD_TUNNEL_ENDPOINT
if [ $AVAILABLE_PORT ]; then
    echo "## Found available port: $AVAILABLE_PORT"
else
    echo "## No available port found!"
    exit 1
fi
DBPORT=$AVAILABLE_PORT
BIND_ADDRESS=127.0.0.1

# Set up tunnel
ssh -o StrictHostKeyChecking=no -x -a -4 -T -N -L $BIND_ADDRESS:$DBPORT:$DBURL $FORWARD_TUNNEL_ENDPOINT &

# Kill ssh process when bootstrapper dies, to prevent lingering ssh's
trap 'jobs -p | xargs kill' EXIT

# Overwrite DBURL
DBURL=$BIND_ADDRESS:$DBPORT

fi

If PYTHON was not set as an argument, detect it here.

if [[ -z $PYTHON ]]; then PYTHON=which python fi

Reuse existing VE if specified

if [[ $GLOBAL_VIRTENV ]]; then

VIRTENV=$GLOBAL_VIRTENV

# activate the virtualenv
source $VIRTENV/bin/activate

# we never clean up global virtualenvs -- remove the 'v' cleanup flag
CLEANUP=$(echo $CLEANUP | tr -d 'v')

# this assumes that the VE lives outside of the pilot sandbox, which MUST be
# true, as at the point where a global VE can be specified, the pilot UID is
# still unknown.  That only conflicts if the pilot sandbox is specified
# explicitly, and the global VE lives therein.  This case is, at this point,
# ignored.

else

bootstrap virtualenv at default location

VIRTENV=$SANDBOX/virtualenv/
# create/update virtualenv.  This activates it.
installvenv $VIRTENV

fi

check if creation succeeded

if [[ ! -d $VIRTENV || ! -f $VIRTENV/bin/activate ]]; then echo "Virtual Environment at $VIRTENV not found, install or upgrade failed. Continue anyways."

in the rare case that everything is already installed in system space, we

# actually don't need a virtualenv, and thus continue here.

fi

-----------------------------------------------------------------------------

launch the radical agent

# AGENT_CMD="python radical-pilot-agent.py\ -b $BENCHMARK\ -c $CORES\ -d $DEBUG\ -j $TASK_LAUNCH_METHOD\ -k $MPI_LAUNCH_METHOD\ -l $LRMS\ -m $DBURL\ -n $DBNAME\ -p $PILOTID\ -s $SESSIONID\ -t $RUNTIME\ -v $VERSION"

echo "" echo "################################################################################" echo "## Launching radical-pilot-agent for $CORES cores." echo "## CMDLINE: $AGENT_CMD" $AGENT_CMD AGENT_EXITCODE=$?

cleanup flags:

l : pilot log files

u : unit work dirs

v : virtualenv

e : everything

echo "CLEANUP: $CLEANUP" contains $CLEANUP 'l' && echo "rm -r $SANDBOX/AGENT." contains $CLEANUP 'u' && echo "rm -r $SANDBOX/unit-" contains $CLEANUP 'v' && echo "rm -r $VIRTENV/" contains $CLEANUP 'e' && echo "rm -r $SANDBOX/"

contains $CLEANUP 'l' && rm -r $SANDBOX/AGENT.*

contains $CLEANUP 'u' && rm -r $SANDBOX/unit-*

contains $CLEANUP 'v' && rm -r $VIRTENV/

contains $CLEANUP 'e' && rm -r $SANDBOX/

... and exit

exit $AGENT_EXITCODE

vivek-bala commented 9 years ago

These are the agent files within the pilot folder (pilot-542d6ea8f8cdba3e87873fd3). Could you paste the STDOUT, STDERR and shell script within the compute unit folder unit-542d6f29f8cdba3e87873fdd (cd pilot-542d6ea8f8cdba3e87873fd3/unit-542d6f29f8cdba3e87873fdd )

ashkurti commented 9 years ago

Sorry ... here are the right ones: STDERR: Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in from extasy import pcz from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz from extasy import pcz from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz Traceback (most recent call last): File "pycoco.py", line 2, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from scipy.linalg import from scipy.linalg import from scipy.linalg import from scipy.linalg import from scipy.linalg import from scipy.linalg import from scipy.linalg import from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import from scipy.linalg import from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import from .misc import from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from . import blas from . import blas from . import blas from . import blas from . import blas from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas from scipy.linalg import _fblas from scipy.linalg import _fblas from scipy.linalg import _fblas from scipy.linalg import _fblas from scipy.linalg import _fblas ImportErrorImportErrorImportError : : : /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportErrorfrom scipy.linalg import _fblas ImportError/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_donefrom scipy.linalg import _fblas from scipy.linalg import _fblas from scipy.linalg import _fblas : from scipy.linalg import _fblas

ImportErrorfrom scipy.linalg import _fblas from scipy.linalg import _fblas

/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done: : ImportErrorfrom scipy.linalg import _fblas ImportError /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_doneImportErrorImportErrorfrom scipy.linalg import _fblas /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_doneImportError : ImportError: : : : /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_doneImportError/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done

: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done : /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_don e/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done

/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done

ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 15, pid: 44369) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 4, pid: 44358) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 44354) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 1, pid: 44355) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 2, pid: 44356) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 3, pid: 44357) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 5, pid: 44359) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 6, pid: 44360) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 7, pid: 44361) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 8, pid: 44362) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 9, pid: 44363) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 10, pid: 44364) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 11, pid: 44365) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 12, pid: 44366) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 13, pid: 44367) exited with status 1 [c545-004.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 14, pid: 44368) exited with status 1 Traceback (most recent call last): File "pycoco.py", line 2, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from scipy.linalg import from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from .misc import from extasy import pcz Traceback (most recent call last): File "pycoco.py", line 2, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import from scipy.linalg import _fblas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas from .misc import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import from scipy.linalg import _fblas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from .misc import from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: from scipy.linalg import _fblas /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: from scipy.linalg import _fblas /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done ImportError from scipy.linalg import _fblas : /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz Traceback (most recent call last): File "pycoco.py", line 2, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import from .misc import from .misc import from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import _fblas from scipy.linalg import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import _fblas from .misc import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in Traceback (most recent call last): File "pycoco.py", line 2, in File "pycoco.py", line 2, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz Traceback (most recent call last): File "pycoco.py", line 2, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from extasy import pcz File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import from .misc import from .misc import from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from . import blas from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from .misc import from scipy.linalg import _fblas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from .misc import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas

ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done from scipy.linalg import _fblas from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done

ashkurti commented 9 years ago

I have followed the link https://github.com/radical-cybertools/ExTASY/blob/devel/docs/scipy_installation_stampede_python_2_7_3.md to install scipy, lapack and blas.

I have done this after loading the python and intel/13.0.2.146 modules.

ashkurti commented 9 years ago

STDOUT login3.stampede(19)$ cat STDOUT TACC: Starting up job 4208657 TACC: Setting up parallel environment for MVAPICH2+mpispawn. TACC: Starting parallel tasks... TACC: MPI job exited with code: 1

TACC: Shutdown complete. Exiting. creating new crd files...

The shell script: login3.stampede(21)$ more radical_pilot_cu_launch_script-kl0U8G.sh

!/bin/bash -l

cd /home1/02998/ardi/radical.pilot.sandbox/pilot-542d6ea8f8cdba3e87873fd3/unit-542d6f29f8cdba3e87873fdd module load intel/13.0.2.146 module load python module load netcdf/4.3.2 module load hdf5/1.8.13 module load amber

/usr/local/bin/ibrun -n 64 -o 0 /bin/bash "-l" "-c" "python pycoco.py --grid 5 --dims 3 --frontpoints 8 --topfile penta.top --mdfile *.ncdf --output pentaopt0" python postexec.py 8 0

ashkurti commented 9 years ago

Do you know which module should I load on stampede for the Fortran compiler ... if the blas problem relates to a fortran compiler ...

vivek-bala commented 9 years ago

Hmmm.. this is new .. seems to be an issue with the scipy installation on stampede .. could you load the python module on stampede and try to import extasy module to see if the error persists ?

$ module load python/2.7.3
$ python
> from extasy import pcz
ashkurti commented 9 years ago

login3.stampede(23)$ module load python login3.stampede(24)$ python --version Python 2.7.3 -- EPD 7.3-2 (64-bit) login3.stampede(25)$ python Enthought Python Distribution -- www.enthought.com Version: 7.3-2 (64-bit)

Python 2.7.3 |EPD 7.3-2 (64-bit)| (default, Apr 11 2012, 17:52:16) [GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2 Type "credits", "demo" or "enthought" for more information.

from extasy import pcz Traceback (most recent call last): File "", line 1, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/pcz.py", line 7, in from scipy.linalg import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/init.py", line 160, in from .misc import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/misc.py", line 5, in from . import blas File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/blas.py", line 155, in from scipy.linalg import _fblas ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/linalg/_fblas.so: undefined symbol: _gfortran_st_write_done

vivek-bala commented 9 years ago

I don't have much experience with Fortran, so I'm not sure what the error is pointing to. Could you try to install scipy again ? and try to import the extasy module on stampede and see if the above error persits ?

vivek-bala commented 9 years ago

By reinstall meant, all of cython, numpy, lapack and BLAS as well.

vivek-bala commented 9 years ago

Since the error seems to emerge from one of the blas files.

vivek-bala commented 9 years ago

Also before reinstalling, please clear everything in /home1/02998/ardi/.local/lib/python2.7/site-packages/

ashkurti commented 9 years ago

I remember that when I installed lapack it did not find a fortran compiler but went on with the c compiler. Now that we now it needs to find the fortran compiler as well, what module should I load on Stampede to have the gfortran compiler ...

ashkurti commented 9 years ago

or what modules do you load on stampede before installing scipy (including cython, blas etc) ...

vivek-bala commented 9 years ago

Just intel/13.0.2.146 and python/2.7.3 On Oct 2, 2014 12:39 PM, "ashkurti" notifications@github.com wrote:

or what modules do you load on stampede before installing scipy (including cython, blas etc) ...

— Reply to this email directly or view it on GitHub https://github.com/radical-cybertools/ExTASY/issues/65#issuecomment-57658335 .

vivek-bala commented 9 years ago

module load gcc/4.7.1 would load gfortran as well.

ashkurti commented 9 years ago

I reinstalled everything. Now I can load pcz without any problems. However, retrying to execute the Coco/Amber workflow, the previous error persists:

[ExTASY-toolsOct2] ardita@poirot 168% extasy --RPconfig stampede.rcfg --Kconfig cocoamber.wcfg Session UID: 542d8a6df8cdba21b08ec161 Pilot UID : 542d8a71f8cdba21b08ec163 [Callback]: ComputePilot '542d8a71f8cdba21b08ec163' state changed to Launching. Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/gromacs.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/amber.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/mmpbsa.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/test.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/coco.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/lsdmap.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/sleep.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/namd.json Cycle 0 Starting Simulation [Callback]: ComputeUnit '542d8a81f8cdba21b08ec168' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec166' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec167' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16b' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16c' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec165' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16a' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec169' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec168' state changed to StagingInput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec166' state changed to StagingInput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec166' state changed to PendingExecution. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec167' state changed to StagingInput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16b' state changed to StagingInput. [Callback]: ComputePilot '542d8a71f8cdba21b08ec163' state changed to PendingActive. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec168' state changed to PendingExecution. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16c' state changed to StagingInput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec165' state changed to StagingInput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec167' state changed to PendingExecution. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16a' state changed to StagingInput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec169' state changed to StagingInput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16b' state changed to PendingExecution. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16c' state changed to PendingExecution. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec165' state changed to PendingExecution. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16a' state changed to PendingExecution. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec169' state changed to PendingExecution. [Callback]: ComputePilot '542d8a71f8cdba21b08ec163' state changed to Active. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec168' state changed to Scheduling. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec166' state changed to Scheduling. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec168' state changed to Executing. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec166' state changed to Executing. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec167' state changed to Scheduling. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec167' state changed to Executing. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16b' state changed to Scheduling. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16b' state changed to Executing. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16c' state changed to Scheduling. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16c' state changed to Executing. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec165' state changed to Scheduling. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec165' state changed to Executing. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16a' state changed to Scheduling. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16a' state changed to Executing. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec169' state changed to Scheduling. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec169' state changed to Executing. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec166' state changed to StagingOutput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec167' state changed to StagingOutput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec165' state changed to StagingOutput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec169' state changed to StagingOutput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec168' state changed to StagingOutput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16b' state changed to StagingOutput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16c' state changed to StagingOutput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16a' state changed to StagingOutput. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec166' state changed to Done. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec167' state changed to Done. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec169' state changed to Done. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec165' state changed to Done. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16a' state changed to Done. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec168' state changed to Done. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16b' state changed to Done. [Callback]: ComputeUnit '542d8a81f8cdba21b08ec16c' state changed to Done. Total Simulation Time : 128.01137805 Simulation Execution Time : Starting Analysis Submitting COCO Compute Unit [Callback]: ComputeUnit '542d8b01f8cdba21b08ec16d' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8b01f8cdba21b08ec16d' state changed to StagingInput. [Callback]: ComputeUnit '542d8b01f8cdba21b08ec16d' state changed to PendingExecution. [Callback]: ComputeUnit '542d8b01f8cdba21b08ec16d' state changed to Scheduling. [Callback]: ComputeUnit '542d8b01f8cdba21b08ec16d' state changed to Executing. [Callback]: ComputeUnit '542d8b01f8cdba21b08ec16d' state changed to StagingOutput. [Callback]: ComputeUnit '542d8b01f8cdba21b08ec16d' state changed to Failed. Log: Output transfer failed: File does not exist: '/home1/02998/ardi/radical.pilot.sandbox/pilot-542d8a71f8cdba21b08ec163/unit-542d8b01f8cdba21b08ec16d/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/output_file_transfer_worker.py", line 148, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '/home1/02998/ardi/radical.pilot.sandbox/pilot-542d8a71f8cdba21b08ec163/unit-542d8b01f8cdba21b08ec16d/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

Analysis Execution time : Cycle 1 Starting Simulation [Callback]: ComputeUnit '542d8b27f8cdba21b08ec16f' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec174' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec16e' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec170' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec171' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec173' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec172' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec175' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec16f' state changed to StagingInput. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec16e' state changed to StagingInput. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec16f' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min11.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min11.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d8b27f8cdba21b08ec170' state changed to StagingInput. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec16e' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d8b27f8cdba21b08ec171' state changed to StagingInput. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec170' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min12.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min12.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d8b27f8cdba21b08ec172' state changed to StagingInput. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec171' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min13.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min13.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d8b27f8cdba21b08ec173' state changed to StagingInput. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec174' state changed to StagingInput. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec172' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min14.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min14.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d8b27f8cdba21b08ec173' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min15.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min15.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d8b27f8cdba21b08ec175' state changed to StagingInput. [Callback]: ComputeUnit '542d8b27f8cdba21b08ec174' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min16.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min16.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '542d8b27f8cdba21b08ec175' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min17.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min17.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

Total Simulation Time : 33.6356990337 Simulation Execution Time : Starting Analysis Submitting COCO Compute Unit [Callback]: ComputeUnit '542d8b48f8cdba21b08ec176' state changed to PendingInputStaging. [Callback]: ComputeUnit '542d8b48f8cdba21b08ec176' state changed to StagingInput. [Callback]: ComputeUnit '542d8b48f8cdba21b08ec176' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

Analysis Execution time : [Callback]: ComputePilot '542d8a71f8cdba21b08ec163' state changed to Canceled. Exception in thread OutputFileTransferWorker-1 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/output_file_transfer_worker.py", line 204, in run File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1058, in next File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1002, in _refresh File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 915, in send_message File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/mongo_client.py", line 1207, in _send_message_with_response <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error' Exception in thread InputFileTransferWorker-2 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in bootstrap_inner File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 81, in run File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/collection.py", line 1677, in find_and_modify File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 445, in command File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 345, in _command File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1058, in next File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1002, in _refresh File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 915, in send_message File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/mongo_client.py", line 1207, in _send_message_with_response <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error' Exception in thread InputFileTransferWorker-1 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib64/python2.6/threading.py", line 532, in bootstrap_inner File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 194, in run File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/collection.py", line 877, in find <type 'exceptions.TypeError'>: 'NoneType' object is not callable

vivek-bala commented 9 years ago

Hmmm .. could you post the contents of STDERR from the failed CoCo compute-unit folder unit-542d8b01f8cdba21b08ec16d ?

ashkurti commented 9 years ago

login3.stampede(125)$ cd ../unit-542d8b01f8cdba21b08ec16d login3.stampede(126)$ ks -bash: ks: command not found login3.stampede(127)$ ls leap.log md_0_0.ncdf md_0_1.ncdf md_0_2.ncdf md_0_3.ncdf md_0_4.ncdf md_0_5.ncdf md_0_6.ncdf md_0_7.ncdf penta.top postexec.py pycoco.py radical_pilot_cu_launch_script-9gpLv2.sh STDERR STDOUT login3.stampede(128)$ more STDERR Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer Traceback (most recent call last): File "pycoco.py", line 5, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer Traceback (most recent call last): File "pycoco.py", line 5, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from scipy import spatial,cluster from scipy import spatial,cluster from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster from scipy import spatial,cluster from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster from scipy import spatial,cluster from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster from scipy import spatial,cluster from scipy import spatial,cluster from scipy import spatial,cluster from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in File "pycoco.py", line 5, in File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in File "pycoco.py", line 5, in File "pycoco.py", line 5, in File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in from extasy import optimizer from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer Traceback (most recent call last): File "pycoco.py", line 5, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer from .qhull import from .qhull import from .qhull import from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done
from .qhull import ImportErrorfrom .qhull import : ImportErrorImportError/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done: : /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_don e

ImportError from .qhull import from .qhull import from .qhull import from .qhull import ImportErrorfrom .qhull import * : ImportError/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done: ImportErrorI File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer

/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_doneImportError : /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from extasy import optimizer from .qhull import from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster from .qhull import from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import from scipy import spatial,cluster ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import from scipy import spatial,cluster ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import from scipy import spatial,cluster ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import from scipy import spatial,cluster ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster from .qhull import from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import ImportError : from .qhull import /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done ImportError from scipy import spatial,cluster : /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError from .qhull import : /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 9, pid: 33326) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 2, pid: 33319) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 33, pid: 25872) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 33317) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 34, pid: 25873) exited with status 1 Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): Traceback (most recent call last): File "pycoco.py", line 5, in Traceback (most recent call last): File "pycoco.py", line 5, in File "pycoco.py", line 5, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer Traceback (most recent call last): File "pycoco.py", line 5, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in from extasy import optimizer File "/home1/02998/ardi/.local/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg/extasy/optimizer.py", line 3, in [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 1, pid: 33318) exited with status 1 from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 32, pid: 25871) exited with status 1 /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from scipy import spatial,cluster ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import from scipy import spatial,cluster ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import from scipy import spatial,cluster ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import from scipy import spatial,cluster ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done
from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from scipy import spatial,cluster File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError from scipy import spatial,cluster : /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done File "/home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/init.py", line 91, in from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done from .qhull import * ImportError: /home1/02998/ardi/.local/lib/python2.7/site-packages/scipy/spatial/qhull.so: undefined symbol: _gfortran_st_write_done [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 3, pid: 33320) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 16, pid: 27576) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 35, pid: 25874) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 4, pid: 33321) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 17, pid: 27577) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 36, pid: 25875) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 5, pid: 33322) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 18, pid: 27578) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 37, pid: 25876) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 6, pid: 33323) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 19, pid: 27579) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 38, pid: 25877) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 7, pid: 33324) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 20, pid: 27580) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 39, pid: 25878) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 8, pid: 33325) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 21, pid: 27581) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 40, pid: 25879) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 10, pid: 33327) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 22, pid: 27582) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 41, pid: 25880) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 11, pid: 33328) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 23, pid: 27583) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 42, pid: 25881) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 12, pid: 33329) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 24, pid: 27584) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 43, pid: 25882) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 13, pid: 33330) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 25, pid: 27585) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 14, pid: 33331) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 59, pid: 28488) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 48, pid: 28477) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 26, pid: 27586) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 45, pid: 25884) exited with status 1 [c454-901.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 15, pid: 33332) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 49, pid: 28478) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 27, pid: 27587) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 46, pid: 25885) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 50, pid: 28479) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 28, pid: 27588) exited with status 1 [c550-603.stampede.tacc.utexas.edu:mpispawn_2][child_handler] MPI process (rank: 47, pid: 25886) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 51, pid: 28480) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 30, pid: 27590) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 52, pid: 28481) exited with status 1 [c550-602.stampede.tacc.utexas.edu:mpispawn_1][child_handler] MPI process (rank: 31, pid: 27591) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 53, pid: 28482) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 54, pid: 28483) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 55, pid: 28484) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 56, pid: 28485) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 57, pid: 28486) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 58, pid: 28487) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 60, pid: 28489) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 61, pid: 28490) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 62, pid: 28491) exited with status 1 [c550-604.stampede.tacc.utexas.edu:mpispawn_3][child_handler] MPI process (rank: 63, pid: 28492) exited with status 1

vivek-bala commented 9 years ago

I reinstalled everything. Now I can load pcz without any problems.

Sorry, wasn't sure what you meant, did the installation with gfortran fix the issue or did you reinstall again as per https://github.com/radical-cybertools/ExTASY/blob/devel/docs/scipy_installation_stampede_python_2_7_3.md ?

ashkurti commented 9 years ago

I reinstalled all modules specified at https://github.com/radical-cybertools/ExTASY/blob/devel/docs/scipy_installation_stampede_python_2_7_3.md.

vivek-bala commented 9 years ago

Hmmm, so pcz can get imported but not optimizer. Not sure why the error is coming up. I will try to reproduce and debug.

ashkurti commented 9 years ago

Thanks a lot!

ashkurti commented 9 years ago

Retrying today I get the following error:

[ExTASY-toolsOct2] ardita@poirot 179% extasy --RPconfig stampede.rcfg --Kconfig cocoamber.wcfg Session UID: 542e76aef8cdba2215198d14 Pilot UID : 542e76b1f8cdba2215198d16 Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/gromacs.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/amber.json [Callback]: ComputePilot '542e76b1f8cdba2215198d16' state changed to Launching. Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/mmpbsa.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/test.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/coco.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/lsdmap.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/sleep.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/namd.json Cycle 0 Starting Simulation [Callback]: ComputeUnit '542e76b8f8cdba2215198d19' state changed to PendingInputStaging. [Callback]: ComputeUnit '542e76b8f8cdba2215198d1f' state changed to PendingInputStaging. [Callback]: ComputeUnit '542e76b8f8cdba2215198d1a' state changed to PendingInputStaging. [Callback]: ComputeUnit '542e76b8f8cdba2215198d1b' state changed to PendingInputStaging. [Callback]: ComputeUnit '542e76b8f8cdba2215198d18' state changed to PendingInputStaging. [Callback]: ComputeUnit '542e76b8f8cdba2215198d1c' state changed to PendingInputStaging. [Callback]: ComputeUnit '542e76b8f8cdba2215198d1d' state changed to PendingInputStaging. [Callback]: ComputeUnit '542e76b8f8cdba2215198d1e' state changed to PendingInputStaging. [Callback]: ComputeUnit '542e76b8f8cdba2215198d19' state changed to StagingInput. [Callback]: ComputeUnit '542e76b8f8cdba2215198d1f' state changed to StagingInput. [Callback]: ComputePilot '542e76b1f8cdba2215198d16' state changed to Failed. pilot manager controller thread caught system exit -- forcing application shutdown Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/bin/extasy", line 9, in load_entry_point('radical.ensemblemd.extasy==0.1', 'console_scripts', 'extasy')() File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/extasy/bin/runme.py", line 130, in main Simulator(umgr,RPconfig,Kconfig,i) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/extasy/bin/Simulator/Amber/simulator.py", line 55, in Simulator umgr.wait_units() File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/unit_manager.py", line 669, in wait_units time.sleep (0.1) KeyboardInterrupt

ashkurti commented 9 years ago

Retrying again today with active debug I get the following output (last lines) very similar to what I get for the GROMACS/LSDMap workflow:

2014:10:06 11:38:11 radical.pilot.MainProcess: [DEBUG ] read : [ 23] [ 55](Shared connection to stampede.tacc.utexas.edu closed.n) 2014:10:06 11:38:11 radical.pilot.MainProcess: [ERROR ] Pilot launching failed: Insufficient system resources: Insufficient system resources: read from process failed '[Errno 5] Input/output error' : (Shared connection to stampede.tacc.utexas.edu closed. ) ((Shared connection to stampede.tacc.utexas.edu closed. )) (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/utils/pty_exceptions.py +59 (translate_exception) : e = se.NoSuccess ("Insufficient system resources: %s" % cmsg)) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/pilot_launcher_worker.py", line 372, in run js = saga.job.Service(job_service_url, session=self._session) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/job/service.py", line 115, in init url, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/slurm/slurm_job.py", line 287, in init_instance self._open () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/slurm/slurm_job.py", line 358, in _open self._logger) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/utils/pty_shell.py", line 228, in init self.pty_shell = self.factory.run_shell (self.pty_info) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/utils/pty_shell_factory.py", line 418, in run_shell self._initialize_pty (sh_slave, info, is_shell=True) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/utils/pty_shell_factory.py", line 384, in _initialize_pty raise ptye.translate_exception (e) NoSuccess: Insufficient system resources: Insufficient system resources: read from process failed '[Errno 5] Input/output error' : (Shared connection to stampede.tacc.utexas.edu closed. ) ((Shared connection to stampede.tacc.utexas.edu closed. )) (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/utils/pty_exceptions.py +59 (translate_exception) : e = se.NoSuccess ("Insufficient system resources: %s" % cmsg))

2014:10:06 11:38:11 radical.pilot.MainProcess: [DEBUG ] Transferring input file file://localhost//users/ardita/coam-on-stampede/mdshort.in -> sftp://stampede.tacc.utexas.edu/home1/02998/ardi/radical.pilot.sandbox/pilot-543270fc54737d754bb10c52//unit-5432710354737d754bb10c56/mdshort.in 2014:10:06 11:38:11 radical.pilot.MainProcess: [DEBUG ] saga.fs.File ('file://localhost//users/ardita/coam-on-stampede/mdshort.in') 2014:10:06 11:38:11 radical.pilot.MainProcess: [DEBUG ] write: [ 22] [ 167](mput "//users/ardita/coam-on-stampede/mdshort.in" "/home1/02998/ardi/radical.pilot.sandbox/pilot-543270fc54737d754bb10c52/unit-5432710354737d754bb10c56/mdshort.in" nn) 2014:10:06 11:38:11 radical.pilot.MainProcess: [DEBUG ] read : [ 22] [ 171](mput "//users/ardita/coam-on-stampede/mdshort.in" "/home1/02998/ardi/radical.pilot.sandbox/pilot-543270fc54737d754bb10c52/unit-5432710354737d754bb10c56/mdshort.in" n) 2014:10:06 11:38:11 radical.pilot.MainProcess: [INFO ] ComputePilot '543270fc54737d754bb10c52' state changed from 'Launching' to 'Failed'. [Callback]: ComputePilot '543270fc54737d754bb10c52' state changed to Failed. pilot manager controller thread caught system exit -- forcing application shutdown 2014:10:06 11:38:11 radical.pilot.MainProcess: [DEBUG ] read : [ 19] [ 79](//users/ardita/coam-on-stampede/mdshort.in 0% 0 0.0KB/s --:-- ETA) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/bin/extasy", line 9, in load_entry_point('radical.ensemblemd.extasy==0.1', 'console_scripts', 'extasy')() File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/extasy/bin/runme.py", line 130, in main Simulator(umgr,RPconfig,Kconfig,i) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/extasy/bin/Simulator/Amber/simulator.py", line 55, in Simulator umgr.wait_units() File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/unit_manager.py", line 669, in wait_units time.sleep (0.1) KeyboardInterrupt

vivek-bala commented 9 years ago

Not sure why you would get insufficient system resources. Could you check the disk quota on Stampede ? You would get this as soon as you ssh into stampede. It would look like 

------------------------ Disk quotas for user vivek91 -------------------------
| Disk         Usage (GB)     Limit    %Used   File Usage       Limit   %Used |
| /home1              2.1       5.0    41.74        93099      150000   62.07 |
| /work               6.4    1024.0     0.63        70397     3000000    2.35 |
-------------------------------------------------------------------------------
ashkurti commented 9 years ago

Sure:

-------------------------- Disk quotas for user ardi --------------------------
| Disk         Usage (GB)     Limit    %Used   File Usage       Limit   %Used |
| /home1              1.0       5.0    20.50        26934      150000   17.96 |
| /work               0.0    1024.0     0.00         2665     3000000    0.09 |
-------------------------------------------------------------------------------
vivek-bala commented 9 years ago

So it shouldn't really show insufficient resources. Does this error persist while running simple_bot.py from radical pilot (https://github.com/radical-cybertools/radical.pilot/blob/master/examples/tutorial/simple_bot.py) on Stampede remotely ?

ashkurti commented 9 years ago

I just retried and the simple_bot.py runs fine remotely (linux/stampede). Checking on the resources though, in the simple_bot.py case we have:

pdesc.runtime  = 10 # minutes
pdesc.cores    = 1
oleweidner commented 9 years ago

Hi Ardita, does this problem still persist with the latest incarnation of ExTASY?

oleweidner commented 9 years ago

... possibly also a resource layer / PTY problem:

2014:10:06 11:38:11 radical.pilot.MainProcess: [ERROR ] Pilot launching failed: Insufficient system resources: Insufficient system resources: read from process failed '[Errno 5] Input/output error' : (Shared connection to stampede.tacc.utexas.edu closed.
) ((Shared connection to stampede.tacc.utexas.edu closed.
oleweidner commented 9 years ago

duplicate of #70. closing