NetCDF: Unknown file formatNetCDF

[ashkurti]

While trying this workflow/platform combination I get the following error:

<<< 2014:10:06 12:55:47 radical.pilot.MainProcess: [ERROR ] Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

2014:10:06 12:55:47 radical.pilot.MainProcess: [DEBUG ] Compute unit 5432834054737d34503804a3 in state StagingInput 2014:10:06 12:55:47 radical.pilot.MainProcess: [DEBUG ] Compute unit 5432834054737d34503804a3 in state StagingInput 2014:10:06 12:55:47 radical.pilot.MainProcess: [DEBUG ] Compute unit 5432834054737d34503804a3 in state StagingInput 2014:10:06 12:55:47 radical.pilot.MainProcess: [INFO ] RUN ComputeUnit '5432834054737d34503804a3' state changed from 'StagingInput' to 'Failed'. [Callback]: ComputeUnit '5432834054737d34503804a3' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[vivek-bala]

This seems like the 2nd cycle of iterations. Did any compute units fail in the previous iteration ?

[ashkurti]

Ok, this is the complete output without the DEBUG:

[ExTASY-toolsOct2] ardita@tirith 159% extasy --RPconfig archer.rcfg --Kconfig cocoamber.wcfg Session UID: 5432858454737d5343153779 Pilot UID : 5432858754737d534315377b Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/gromacs.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/amber.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/mmpbsa.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/test.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/coco.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/lsdmap.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/sleep.json Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/namd.json Cycle 0 Starting Simulation [Callback]: ComputePilot '5432858754737d534315377b' state changed to Launching. [Callback]: ComputeUnit '5432858a54737d534315377e' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432858a54737d534315377f' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432858a54737d5343153780' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432858a54737d534315377d' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432858a54737d5343153783' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432858a54737d5343153782' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432858a54737d5343153784' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432858a54737d5343153781' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432858a54737d534315377e' state changed to StagingInput. [Callback]: ComputeUnit '5432858a54737d534315377f' state changed to StagingInput. [Callback]: ComputeUnit '5432858a54737d5343153780' state changed to StagingInput. [Callback]: ComputeUnit '5432858a54737d534315377e' state changed to PendingExecution. [Callback]: ComputeUnit '5432858a54737d534315377d' state changed to StagingInput. [Callback]: ComputeUnit '5432858a54737d5343153783' state changed to StagingInput. [Callback]: ComputeUnit '5432858a54737d534315377f' state changed to PendingExecution. [Callback]: ComputeUnit '5432858a54737d5343153782' state changed to StagingInput. [Callback]: ComputeUnit '5432858a54737d5343153784' state changed to StagingInput. [Callback]: ComputeUnit '5432858a54737d5343153780' state changed to PendingExecution. [Callback]: ComputeUnit '5432858a54737d5343153781' state changed to StagingInput. [Callback]: ComputeUnit '5432858a54737d534315377d' state changed to PendingExecution. [Callback]: ComputeUnit '5432858a54737d5343153783' state changed to PendingExecution. [Callback]: ComputeUnit '5432858a54737d5343153782' state changed to PendingExecution. [Callback]: ComputeUnit '5432858a54737d5343153784' state changed to PendingExecution. [Callback]: ComputeUnit '5432858a54737d5343153781' state changed to PendingExecution. [Callback]: ComputePilot '5432858754737d534315377b' state changed to PendingActive. [Callback]: ComputePilot '5432858754737d534315377b' state changed to Active. [Callback]: ComputeUnit '5432858a54737d534315377e' state changed to Scheduling. [Callback]: ComputeUnit '5432858a54737d534315377f' state changed to Scheduling. [Callback]: ComputeUnit '5432858a54737d534315377e' state changed to Executing. [Callback]: ComputeUnit '5432858a54737d534315377f' state changed to Executing. [Callback]: ComputeUnit '5432858a54737d5343153780' state changed to Scheduling. [Callback]: ComputeUnit '5432858a54737d5343153780' state changed to Executing. [Callback]: ComputeUnit '5432858a54737d534315377d' state changed to Scheduling. [Callback]: ComputeUnit '5432858a54737d534315377d' state changed to Executing. [Callback]: ComputeUnit '5432858a54737d5343153783' state changed to Scheduling. [Callback]: ComputeUnit '5432858a54737d5343153783' state changed to Executing. [Callback]: ComputeUnit '5432858a54737d5343153782' state changed to Scheduling. [Callback]: ComputeUnit '5432858a54737d5343153782' state changed to Executing. [Callback]: ComputeUnit '5432858a54737d5343153784' state changed to Scheduling. [Callback]: ComputeUnit '5432858a54737d5343153784' state changed to Executing. [Callback]: ComputeUnit '5432858a54737d5343153781' state changed to Scheduling. [Callback]: ComputeUnit '5432858a54737d5343153781' state changed to Executing. [Callback]: ComputeUnit '5432858a54737d534315377e' state changed to StagingOutput. [Callback]: ComputeUnit '5432858a54737d534315377e' state changed to Done. [Callback]: ComputeUnit '5432858a54737d534315377f' state changed to StagingOutput. [Callback]: ComputeUnit '5432858a54737d534315377f' state changed to Done. [Callback]: ComputeUnit '5432858a54737d5343153781' state changed to StagingOutput. [Callback]: ComputeUnit '5432858a54737d5343153781' state changed to Done. [Callback]: ComputeUnit '5432858a54737d5343153782' state changed to StagingOutput. [Callback]: ComputeUnit '5432858a54737d5343153782' state changed to Done. [Callback]: ComputeUnit '5432858a54737d534315377d' state changed to StagingOutput. [Callback]: ComputeUnit '5432858a54737d534315377d' state changed to Done. [Callback]: ComputeUnit '5432858a54737d5343153780' state changed to StagingOutput. [Callback]: ComputeUnit '5432858a54737d5343153780' state changed to Done. [Callback]: ComputeUnit '5432858a54737d5343153783' state changed to StagingOutput. [Callback]: ComputeUnit '5432858a54737d5343153783' state changed to Done. [Callback]: ComputeUnit '5432858a54737d5343153784' state changed to StagingOutput. [Callback]: ComputeUnit '5432858a54737d5343153784' state changed to Done. Total Simulation Time : 209.991783142 Simulation Execution Time : Starting Analysis Submitting COCO Compute Unit [Callback]: ComputeUnit '5432865b54737d5343153785' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432865b54737d5343153785' state changed to StagingInput. [Callback]: ComputeUnit '5432865b54737d5343153785' state changed to PendingExecution. [Callback]: ComputeUnit '5432865b54737d5343153785' state changed to Scheduling. [Callback]: ComputeUnit '5432865b54737d5343153785' state changed to Executing. [Callback]: ComputeUnit '5432865b54737d5343153785' state changed to StagingOutput. [Callback]: ComputeUnit '5432865b54737d5343153785' state changed to Failed. Log: Output transfer failed: File does not exist: '/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-5432858754737d534315377b/unit-5432865b54737d5343153785/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/output_file_transfer_worker.py", line 148, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '/fs4/e290/e290/ardi/radical.pilot.sandbox/pilot-5432858754737d534315377b/unit-5432865b54737d5343153785/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

Analysis Execution time : Cycle 1 Starting Simulation [Callback]: ComputeUnit '5432869754737d5343153787' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432869754737d5343153788' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432869754737d534315378c' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432869754737d5343153786' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432869754737d5343153789' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432869754737d534315378a' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432869754737d534315378b' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432869754737d534315378d' state changed to PendingInputStaging. [Callback]: ComputeUnit '5432869754737d5343153787' state changed to StagingInput. [Callback]: ComputeUnit '5432869754737d5343153788' state changed to StagingInput. [Callback]: ComputeUnit '5432869754737d5343153787' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min11.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min11.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '5432869754737d5343153786' state changed to StagingInput. [Callback]: ComputeUnit '5432869754737d5343153788' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min12.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min12.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '5432869754737d5343153789' state changed to StagingInput. [Callback]: ComputeUnit '5432869754737d5343153786' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min10.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '5432869754737d534315378a' state changed to StagingInput. [Callback]: ComputeUnit '5432869754737d534315378b' state changed to StagingInput. [Callback]: ComputeUnit '5432869754737d5343153789' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min13.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min13.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '5432869754737d534315378a' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min14.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min14.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '5432869754737d534315378c' state changed to StagingInput. [Callback]: ComputeUnit '5432869754737d534315378b' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min15.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min15.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '5432869754737d534315378d' state changed to StagingInput. [Callback]: ComputeUnit '5432869754737d534315378c' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min16.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min16.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

[Callback]: ComputeUnit '5432869754737d534315378d' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/min17.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/min17.crd' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

Total Simulation Time : 13.7401340008 Simulation Execution Time : Starting Analysis Submitting COCO Compute Unit [Callback]: ComputeUnit '543286a554737d534315378e' state changed to PendingInputStaging. [Callback]: ComputeUnit '543286a554737d534315378e' state changed to StagingInput. [Callback]: ComputeUnit '543286a554737d534315378e' state changed to Failed. Log: Input transfer failed: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out))) Traceback (most recent call last): File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 146, in run session=self._session File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in init _adaptor, _adaptor_state, _ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in init url, flags, session, ttype=_ttype) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in init self._init_task = self._adaptor.init_instance (adaptor_state, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function return sync_function (self, _args, _kwargs) File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 978, in init_instance self.initialize () File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 1027, in initialize raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)) DoesNotExist: File does not exist: '//users/ardita/coam-on-stampede/md_1_0.ncdf' - (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py +1027 (initialize) : raise saga.DoesNotExist("File does not exist: '%s' - %s" % (self.url.path, out)))

Analysis Execution time : [Callback]: ComputePilot '5432858754737d534315377b' state changed to Canceled.

[vivek-bala]

Ah ok. So the CoCo compute unit 5432865b54737d5343153785 failed. Could you check the content of the compute unit folder, unit-5432865b54737d5343153785, on Archer ? Could you tell me what the STDERR file contains.

[oleweidner]

I just checked and Coco/Amber works fine for me from remote to Archer with the latest version of the 'devel' branch.

[ashkurti]

The STDERR content is at: https://gist.github.com/ashkurti/71841867fa71a4c86edb. Apparently it is not recognizing the NetCDF format? What should I do in such a case ...

[ashkurti]

I will try to reinstall ExTASY again locally ...

[oleweidner]

Hi Ardita, if you do a module list on Archer, what do you see?

[vivek-bala]

Ah ok, you might have an outdated mdkernels. Please reinstall and it should work.

[vivek-bala]

The netcdf was added to the mdkernels in the last commmit.

[ashkurti]

ardi@eslogin003:/work/e290/e290/ardi/radical.pilot.sandbox/pilot-5432858754737d534315377b/unit-5432865b54737d5343153785> module list
Currently Loaded Modulefiles:
  1) modules/3.2.6.7                       10) pmi/5.0.3-1.0000.9981.128.2.ari       19) atp/1.7.2                             28) checkDisk/1.0
  2) eswrap/1.1.0-1.010400.915.0           11) dmapp/7.0.1-1.0501.8315.8.4.ari       20) PrgEnv-cray/5.1.29                    29) bolt/0.5
  3) switch/1.0-1.0501.47124.1.93.ari      12) gni-headers/3.0-1.0501.8317.12.1.ari  21) pbs/12.1.400.132424                   30) serialJobs/1.0
  4) craype-network-aries                  13) xpmem/0.1-2.0501.48424.3.3.ari        22) craype-ivybridge                      31) python/2.7.6
  5) craype/2.1.1                          14) job/1.5.5-0.1_2.0501.48066.2.43.ari   23) cray-mpich/6.3.1                      32) tkdiff/4.2
  6) cce/8.2.6                             15) csa/3.0.0-1_2.0501.47112.1.91.ari     24) packages-archer                       33) nano/2.2.6
  7) cray-libsci/12.2.0                    16) dvs/2.4_0.9.0-1.0501.1672.2.122.ari   25) budgets/1.1                           34) imagemagick/6.8.8-2
  8) udreg/2.3.2-1.0501.7914.1.13.ari      17) alps/5.1.1-2.0501.8507.1.1.ari        26) checkScript/1.1                       35) leave_time/1.0.0
  9) ugni/5.0-1.0501.8253.10.22.ari        18) rca/1.0.0-2.0501.48090.7.46.ari       27) checkQueue/1.0                        36) epcc-tools/1.2

[ashkurti]

I have got the same problem, just retried after reinstalling everything locally ... according to the development branch ...

[ashkurti]

At the STDERR of the computational unit that failed first among other things I found:

  File "netCDF4.pyx", line 1466, in netCDF4.Dataset.__init__ (netCDF4.c:19738)
  File "netCDF4.pyx", line 1466, in netCDF4.Dataset.__init__ (netCDF4.c:19738)
  File "netCDF4.pyx", line 1466, in netCDF4.Dataset.__init__ (netCDF4.c:19738)
  File "netCDF4.pyx", line 1466, in netCDF4.Dataset.__init__ (netCDF4.c:19738)
  File "netCDF4.pyx", line 1466, in netCDF4.Dataset.__init__ (netCDF4.c:19738)
    self.trjfile = netcdf.Dataset(self.filename)
  File "netCDF4.pyx", line 1466, in netCDF4.Dataset.__init__ (netCDF4.c:19738)
RuntimeErrorRuntimeErrorRuntimeError: : : NetCDF: Unknown file formatNetCDF: Unknown file formatNetCDF: Unknown file format

[vivek-bala]

can you list the contents of the shell script which starts as radicalpilot*.sh in the same compute unit folder please ?

[ashkurti]

Sure:

#!/bin/bash -l
cd /fs4/e290/e290/ardi/pilot-5432bf69f8cdba0f358ca8c4/unit-5432c020f8cdba0f358ca8ce
module load python
module load numpy
module load scipy
module load coco
module load netcdf4-python
module load amber

/opt/cray/alps/5.1.1-2.0501.8507.1.1.ari/bin/aprun -n 24 /bin/bash "-l" "-c" "python pycoco.py --grid 5 --dims 3 --frontpoints 8 --topfile penta.top --mdfile *.ncdf --output pentaopt0"
python postexec.py 8 0

[oleweidner]

Does this problem still persist?

[oleweidner]

Not reproducible. Closed before new round of testing.