Missing radical.pilot-0.38.tar.gz file when submitting onto Stampede

[mingtaiha]

I have been trying to run the submit jobs to Stampede to run an AIMES experiment. For some reason, I am missing the radical.pilot-0.38.tar.gz. The problem does not lie with the experiment as Matteo is able to run them as well. I have made sure that I have the right packages for the runs, and the versions are listed below.

RP Version: 0.38 SAGA Version: 0.40 pip Version: 8.0.2

2016-01-22 23:37:39,434: radical.saga.pty    : MainProcess                     : PilotLauncherWorker-1: ERROR   : DoesNotExist: file copy failed: mput  "/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/pilot/radical.pilot-0.38.tar.gz" "/work/03662/tg829619/radical.pilot.sandbox/rp.session.ip-10-184-31-85.mingtha.016822.0010-pilot.0000/radical.pilot-0.38.tar.gz"
stat /home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/pilot/radical.pilot-0.38.tar.gz: No such file or directory
sftp> 
2016-01-22 23:37:39,438: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: ERROR   : Using bootstrapper /home/mingtha/ve/aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh
Copying bootstrapper 'file://localhost/home/mingtha/ve/aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh' to agent sandbox (<saga.filesystem.directory.Directory object at 0x7fba2c0aa050>).
Copying sdist 'file://localhost/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/utils/radical.utils-0.38.tar.gz' to sandbox (sftp://stampede.tacc.utexas.edu/work/03662/tg829619/radical.pilot.sandbox/rp.session.ip-10-184-31-85.mingtha.016822.0010-pilot.0000/).
Copying sdist 'file://localhost/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/saga/saga-python-0.40.tar.gz' to sandbox (sftp://stampede.tacc.utexas.edu/work/03662/tg829619/radical.pilot.sandbox/rp.session.ip-10-184-31-85.mingtha.016822.0010-pilot.0000/).
Copying sdist 'file://localhost/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/pilot/controller/..//radical.pilot-0.38.tar.gz' to sandbox (sftp://stampede.tacc.utexas.edu/work/03662/tg829619/radical.pilot.sandbox/rp.session.ip-10-184-31-85.mingtha.016822.0010-pilot.0000/).
Pilot launching failed! (file copy failed: mput  "/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/pilot/radical.pilot-0.38.tar.gz" "/work/03662/tg829619/radical.pilot.sandbox/rp.session.ip-10-184-31-85.mingtha.016822.0010-pilot.0000/radical.pilot-0.38.tar.gz"
stat /home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/pilot/radical.pilot-0.38.tar.gz: No such file or directory
sftp>  (/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/saga/utils/pty_shell.py +1023 (run_copy_to)  :  raise se.DoesNotExist._log (info['logger'], "file copy failed: %s" % str(out))))
Traceback (most recent call last):
  File "/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/pilot/controller/pilot_launcher_worker.py", line 587, in run
    sandbox_tgt.copy(sdist_url, os.path.basename(str(sdist_url)))
  File "/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/saga/namespace/directory.py", line 353, in copy
    if url_2  :  return self._adaptor.copy (url_1, url_2, flags, ttype=ttype)
  File "/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/saga/adaptors/cpi/decorators.py", line 57, in wrap_function
    return sync_function (self, *args, **kwargs)
  File "/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py", line 556, in copy
    files_copied = copy_shell.stage_to_remote (src.path, tgt.path, rec_flag)
  File "/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/saga/utils/pty_shell.py", line 906, in stage_to_remote
    raise ptye.translate_exception (e)
DoesNotExist: file copy failed: mput  "/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/pilot/radical.pilot-0.38.tar.gz" "/work/03662/tg829619/radical.pilot.sandbox/rp.session.ip-10-184-31-85.mingtha.016822.0010-pilot.0000/radical.pilot-0.38.tar.gz"
stat /home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/pilot/radical.pilot-0.38.tar.gz: No such file or directory
sftp>  (/home/mingtha/ve/aimes-swift-experiments/local/lib/python2.7/site-packages/saga/utils/pty_shell.py +1023 (run_copy_to)  :  raise se.DoesNotExist._log (info['logger'], "file copy failed: %s" % str(out)))

[vivek-bala]

Downgrade your pip to 1.4.1 and try again.

[vivek-bala]

Within your VE pip install pip==1.4.1 --upgrade

[mturilli]

Tried that, does not work anymore. Specifically, radical.utils fails to install. Thanks to Harry Potter I got it working with pip 8.0.2 but Harry told me that he does not like Ming so no joy for him.

[andre-merzky]

Hmm, that is puzzling. Ming, would you please run this and send the output (assuming your virtualenv is in $VE):

pip install --upgrade -v .
du -a $VE/lib/python2.7/site-packages/radical/pilot/

Thanks!

[mingtaiha]

I get the output below. It should be noted that my RP version is now v0.39.RC1@devel:

(aimes-swift-experiments) mingtha@ip-10-184-31-85:~/AIMES_Only_Files_v2$ du -a aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/exceptions.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/logentry.pyc
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/pilot_manager.pyc
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/compute_pilot.pyc
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/worker/agent.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/worker/heartbeat.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/worker/update.py
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/worker/agent.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/worker/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/worker/__init__.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/worker/heartbeat.pyc
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/worker/update.pyc
92  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/worker
140 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/bootstrapper/cacert.pem.gz
52  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/bootstrapper/bootstrap_1.sh
0   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/bootstrapper/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/bootstrapper/__init__.pyc
200 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/bootstrapper
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/session.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/staging_directives.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/states.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/context.py
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/compute_pilot.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/fork.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/runjob.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/aprun.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/base.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/mpiexec.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/fork.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/mpirun_dplace.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/ccmrun.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/dplace.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/ibrun.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/mpirun_rsh.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/dplace.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/ibrun.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/mpirun_rsh.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/mpirun_ccmrun.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/orte.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/poe.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/poe.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/runjob.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/yarn.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/mpirun_dplace.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/__init__.pyc
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/yarn.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/orte.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/mpirun_ccmrun.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/ssh.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/ssh.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/mpirun.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/mpiexec.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/ccmrun.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/mpirun.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/base.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm/aprun.pyc
188 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/lm
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_output/default.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_output/base.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_output/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_output/__init__.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_output/default.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_output/base.pyc
36  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_output
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/base.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/torus.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/torus.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/__init__.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/continuous.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/yarn.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/__init__.pyc
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/yarn.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/scattered.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/scattered.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/continuous.pyc
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler/base.pyc
108 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/scheduler
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/abds.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/base.py
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/shell.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/__init__.py
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/abds.pyc
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/popen.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/yarn.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/__init__.pyc
0   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/yarn.py
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/shell.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/popen.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing/base.pyc
136 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/executing
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_input/default.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_input/base.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_input/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_input/__init__.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_input/default.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_input/base.pyc
36  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/staging_input
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/fork.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/ccm.pyc
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/base.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/slurm.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/slurm.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/fork.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/lsf.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/sge.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/pbspro.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/pbspro.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/lsf.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/__init__.py
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/loadleveler.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/yarn.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/__init__.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/yarn.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/sge.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/torque.py
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/loadleveler.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/torque.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/ccm.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm/base.pyc
144 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/rm
28  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/radical-pilot-spawner.sh
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/__init__.pyc
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/radical-pilot-agent-multicore.pyc
28  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent/radical-pilot-agent-multicore.py
736 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/agent
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler/direct_submission.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler/direct_submission.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler/interface.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler/round_robin.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler/interface.pyc
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler/backfilling.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler/__init__.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler/round_robin.py
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler/backfilling.py
72  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/scheduler
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/pilot_manager.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/resource_config.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/constants.py
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/session.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/db/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/db/__init__.pyc
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/db/database.pyc
28  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/db/database.py
64  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/db
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/constants.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/types.py
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/pilot_manager_controller.pyc
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/unit_manager_controller.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/input_file_transfer_worker.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/output_file_transfer_worker.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/__init__.py
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/unit_manager_controller.pyc
44  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/pilot_launcher_worker.py
28  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/pilot_manager_controller.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/__init__.pyc
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/input_file_transfer_worker.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/output_file_transfer_worker.py
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller/pilot_launcher_worker.pyc
200 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/controller
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/SDIST
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/staging_directives.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/resource_config.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/states.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/logentry.py
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/compute_unit.py
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/compute_unit.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/VERSION
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_18.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_147.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_114.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_262.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_18.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_169.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_163.py
0   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_147.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/__init__.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_114.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_262.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_213.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_163.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_213.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues/issue_169.pyc
76  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/issues
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_session.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_session.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_unit.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_pilot_manager.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_pilot.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_unit_manager.pyc
0   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/__init__.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_unit.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_resource_configs.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_unit_manager.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/__init__.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_pilot.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_resource_configs.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api/test_pilot_manager.pyc
92  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/api
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/__init__.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/remote/test_remote.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/remote/test_remote.py
0   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/remote/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/remote/__init__.pyc
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/remote
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/__init__.pyc
0   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/units/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/units/__init__.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests/units
212 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/tests
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/__init__.pyc
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/compute_pilot_description.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/types.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_aliases.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/agent_simple.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_local.json
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_ncsa.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_iu.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_das4.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_epsrc.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/agent_cray.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_radical.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/agent_default.json
0   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/__init__.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_nersc.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_rice.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_ncar.json
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_futuregrid.json
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_xsede.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_chameleon.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_lrz.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/__init__.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/agent_osg.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_yale.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_stfc.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_osg.json
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs/resource_ornl.json
128 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/configs
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/context.pyc
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/unit_manager.pyc
400 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/radical.pilot-v0.39.RC1-devel.tar.gz
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/compute_pilot_description.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/exceptions.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/compute_unit_description.pyc
28  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/unit_manager.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/compute_unit_description.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/misc.pyc
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/session.py
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/pubsub.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/db_connection_info.pyc
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/db_utils.py
40  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/component.py
12  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/session.pyc
32  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/component.pyc
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/queue.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/__init__.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/db_connection_info.py
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/prof_utils.pyc
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/analysis.py
16  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/pubsub.py
8   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/db_utils.pyc
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/__init__.pyc
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/prof_utils.py
24  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/analysis.pyc
20  aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/queue.py
4   aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils/misc.py
304 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/utils
2776    aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/

[andre-merzky]

what was the output of the pip install --upgrade -v . command?

Thanks

[marksantcroos]

400 aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/radical.pilot-v0.39.RC1-devel.tar.gz

The file is there. You still get the same error when you actually try to use it?

[mingtaiha]

@andre-merzky The output of pip install --upgrade -v . is in the file attached pip_upgrade.txt

@marksantcroos When I try to run the experiment again, I get the following error:

2016-01-23 18:46:10,256: radical.pilot       : MainProcess                     : MainThread     : ERROR   : session create failed
Traceback (most recent call last):
  File "/home/mingtha/AIMES_Only_Files_v2/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/pilot/session.py", line 138, in __init__
    ru.reset_id_counters(prefix=['pmgr', 'umgr', 'pilot', 'unit', 'unit.%(counter)06d'])
AttributeError: 'module' object has no attribute 'reset_id_counters'

Caught exception: Couldn't create new session (database URL 'None' incorrect?): 'module' object has no attribute 'reset_id_counters'
Traceback (most recent call last):
  File "/home/mingtha/AIMES_Only_Files_v2/aimes-swift-experiments/local/lib/python2.7/site-packages/aimes/emgr/emgr.py", line 513, in execute_workload
    session           = rp.Session(database_url=cfg['mongodb'])
  File "/home/mingtha/AIMES_Only_Files_v2/aimes-swift-experiments/local/lib/python2.7/site-packages/radical/pilot/session.py", line 166, in __init__
    % (self._dburl, ex))
PilotException: Couldn't create new session (database URL 'None' incorrect?): 'module' object has no attribute 'reset_id_counters'
Traceback (most recent call last):
  File "aimes_only.py", line 85, in <module>
    aimes.emgr.execute_workload(cfg, run)
  File "/home/mingtha/AIMES_Only_Files_v2/aimes-swift-experiments/local/lib/python2.7/site-packages/aimes/emgr/emgr.py", line 636, in execute_workload
    session.close(cleanup=False, terminate=True)
UnboundLocalError: local variable 'session' referenced before assignment

[andre-merzky]

Ah, that is very different, and now caused by using the devel branch of RP. Please also install the devel branch or radical.utils the same way (or the same tag), and try once more! thanks, Andre.

[mingtaiha]

It seems alright now. I ran on Stampede successfully.

[andre-merzky]

Hmm, so, what has changed ultimately? Did the install from the git clone succeed, but from pypi fail? Could you please try an explicit install from pypi then again (instead of having it pulled as dependency):

pip uninstall -y radical.pilot
pip install --upgrade -v radical.pilot
du -a $VE/lib/python2.7/site-packages/radical/pilot/ | grep tar.gz

[mingtaiha]

I think what changed was when I upgraded my RP version to 0.39.RC1@devel, as well as upgrading radical.utils to the most recent version. To confirm this, I checked by submitting a pilot to the queue (the error occurs before the pilot makes it to the queue it seems) for each RP version installed. The pilot encountered problems only on the 0.38 Version.

Interestingly, when I run pip install --upgrade -v radical.pilot as mentioned above, I install RP Version 0.38, which was causing me problems. When I install from a clone of radical.pilot I pulled yesterday, I install RP Version 0.39.RC1@devel.

When running the command

du -a $VE/lib/python2.7/site-packages/radical/pilot/ | grep tar.gz

on RP 0.38, I get no output, indicating that no such file exists. When I install RP 0.39.RC1, I get the following output

400 ../aimes-swift-experiments/lib/python2.7/site-packages/radical/pilot/radical.pilot-v0.39.RC1-devel.tar.gz

[andre-merzky]

I am quite confused by this :( The setup.py has not been touched in quite a while, and should be the same in the pypi tarball and in the github clone. I am not sure what is going on here... :/

Would you mind checking if installing the master branch from the git clone installs correctly for you?

[andre-merzky]

Has this problem be resolved?

[andre-merzky]

Closing due to lack of feedback