search

radical-cybertools / radical.pilot

RADICAL-Pilot
http://radical-cybertools.github.io/radical-pilot/index.html
Other
54 stars 23 forks source link

RP-devel failing on localhost #737

Closed vivek-bala closed 8 years ago

vivek-bala commented 8 years ago

Used saga-devel and ru-devel.

verbose log:

$ RADICAL_PILOT_VERBOSE=debug python getting_started_local.py 
2015-09-18 16:41:53,183: radical.pilot       : MainProcess                     : MainThread     : INFO    : python.interpreter   version: 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2]
2015-09-18 16:41:53,183: radical.pilot       : MainProcess                     : MainThread     : INFO    :                      pid: 3503
2015-09-18 16:41:53,183: radical.pilot       : MainProcess                     : MainThread     : INFO    :                      tid: MainThread
2015-09-18 16:41:53,183: radical.pilot       : MainProcess                     : MainThread     : INFO    : radical.pilot        version: v0.35-381-ga9e567a@devel
2015-09-18 16:41:53,245: radical.pilot       : MainProcess                     : MainThread     : INFO    : using database mongodb://extasy:extasyproject@extasy-db.epcc.ed.ac.uk/radicalpilot
2015-09-18 16:41:53,924: radical.pilot       : MainProcess                     : MainThread     : INFO    : New Session created: {'database_url': <radical.utils.url.Url object at 0x7f103f64b290>, 'connected': 1442608913.683164, 'uid': 'rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024', 'created': 1442608913.683164}.
2015-09-18 16:41:53,925: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_ornl.json
2015-09-18 16:41:53,936: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for ornl.titan
2015-09-18 16:41:53,939: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_das4.json
2015-09-18 16:41:53,949: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for das4.fs2
2015-09-18 16:41:53,951: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_epsrc.json
2015-09-18 16:41:53,960: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for epsrc.archer
2015-09-18 16:41:53,963: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_nersc.json
2015-09-18 16:41:54,009: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.hopper_orte
2015-09-18 16:41:54,011: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.hopper_ccm
2015-09-18 16:41:54,014: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.edison_ccm
2015-09-18 16:41:54,016: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.edison
2015-09-18 16:41:54,018: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.edison_orte
2015-09-18 16:41:54,020: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.hopper
2015-09-18 16:41:54,022: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_ncsa.json
2015-09-18 16:41:54,039: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for ncsa.bw_orte
2015-09-18 16:41:54,041: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for ncsa.bw_ccm
2015-09-18 16:41:54,043: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for ncsa.bw
2015-09-18 16:41:54,045: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_iu.json
2015-09-18 16:41:54,055: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for iu.bigred2
2015-09-18 16:41:54,057: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for iu.bigred2_ccm
2015-09-18 16:41:54,058: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_local.json
2015-09-18 16:41:54,062: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for local.localhost
2015-09-18 16:41:54,063: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_lrz.json
2015-09-18 16:41:54,067: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for lrz.supermuc
2015-09-18 16:41:54,068: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_aliases.json
2015-09-18 16:41:54,069: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_rice.json
2015-09-18 16:41:54,077: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for rice.biou
2015-09-18 16:41:54,078: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for rice.davinci
2015-09-18 16:41:54,080: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_futuregrid.json
2015-09-18 16:41:54,102: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.xray
2015-09-18 16:41:54,103: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.echo
2015-09-18 16:41:54,104: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.india
2015-09-18 16:41:54,105: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.bravo
2015-09-18 16:41:54,106: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.xray_ccm
2015-09-18 16:41:54,107: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.delta
2015-09-18 16:41:54,108: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_xsede.json
2015-09-18 16:41:54,134: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.blacklight
2015-09-18 16:41:54,136: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.supermic
2015-09-18 16:41:54,137: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.lonestar
2015-09-18 16:41:54,138: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.comet
2015-09-18 16:41:54,139: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.stampede
2015-09-18 16:41:54,140: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.trestles
2015-09-18 16:41:54,141: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.gordon
2015-09-18 16:41:54,142: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_stfc.json
2015-09-18 16:41:54,146: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for stfc.joule
2015-09-18 16:41:54,147: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_ncar.json
2015-09-18 16:41:54,151: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for ncar.yellowstone
2015-09-18 16:41:54,152: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/configs/resource_radical.json
2015-09-18 16:41:54,156: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for radical.tutorial
session id: rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024
2015-09-18 16:41:54,282: radical.pilot       : MainProcess                     : Thread-1       : DEBUG   : Worker thread (ID: Thread-1[139707742017280]) for PilotManager pmgr.0000 started.
2015-09-18 16:41:54,290: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : saga.utils.PTYShell ('fork://localhost/')
2015-09-18 16:41:54,293: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : Connected to MongoDB. Serving requests for PilotManager pmgr.0000.
2015-09-18 16:41:54,851: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : Determined remote working directory for fork://localhost/: '/home/vivek'
2015-09-18 16:41:55,194: radical.pilot       : MainProcess                     : InputFileTransferWorker-1: INFO    : Starting InputFileTransferWorker
2015-09-18 16:41:55,195: radical.pilot       : MainProcess                     : InputFileTransferWorker-1: DEBUG   : Connected to MongoDB. Serving requests for UnitManager umgr.0000.
2015-09-18 16:41:55,196: radical.pilot       : MainProcess                     : InputFileTransferWorker-2: INFO    : Starting InputFileTransferWorker
2015-09-18 16:41:55,196: radical.pilot       : MainProcess                     : InputFileTransferWorker-2: DEBUG   : Connected to MongoDB. Serving requests for UnitManager umgr.0000.
2015-09-18 16:41:55,198: radical.pilot       : MainProcess                     : OutputFileTransferWorker-1: DEBUG   : Connected to MongoDB. Serving requests for UnitManager umgr.0000.
2015-09-18 16:41:55,198: radical.pilot       : MainProcess                     : OutputFileTransferWorker-2: DEBUG   : Connected to MongoDB. Serving requests for UnitManager umgr.0000.
2015-09-18 16:41:55,198: radical.pilot       : MainProcess                     : MainThread     : INFO    : Loaded scheduler: DirectSubmissionScheduler.
2015-09-18 16:41:55,199: radical.pilot       : MainProcess                     : Thread-3       : DEBUG   : Worker thread (ID: Thread-3[139707484722944]) for UnitManager umgr.0000 started.
2015-09-18 16:41:55,839: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: INFO    : Launching ComputePilot pilot.0000
2015-09-18 16:41:55,840: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: INFO    : Read agent config file: /home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/controller/../configs/agent_default.json
2015-09-18 16:41:56,334: radical.pilot       : MainProcess                     : MainThread     : INFO    : Scheduled ComputeUnits [unit.000000 (Scheduling     : /bin/sleep ['1']) (139707750743056)] on ComputePilot 'pilot.0000'.
2015-09-18 16:41:56,334: radical.pilot       : MainProcess                     : MainThread     : INFO    : 0 units remain unscheduled
2015-09-18 16:41:56,418: radical.pilot       : MainProcess                     : InputFileTransferWorker-1: DEBUG   : InputStagingController: unit found: unit.000000
2015-09-18 16:41:56,548: radical.pilot       : MainProcess                     : InputFileTransferWorker-1: DEBUG   : InputStagingController: unit.000000 : push to agent
2015-09-18 16:41:56,618: radical.pilot       : MainProcess                     : Thread-1       : INFO    : ComputePilot 'pilot.0000' state changed from 'PendingLaunch' to 'Launching'.
[Callback]: ComputePilot 'pilot.0000' state: Launching.
2015-09-18 16:41:56,731: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: INFO    : Using bootstrapper /home/vivek/Research/ves/myenv/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh
2015-09-18 16:41:56,731: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : Copying bootstrapper 'file://localhost/home/vivek/Research/ves/myenv/lib/python2.7/site-packages/radical/pilot/bootstrapper/default_bootstrapper.sh' to agent sandbox (<saga.filesystem.directory.Directory object at 0x7f103edaf910>).
2015-09-18 16:41:56,774: radical.pilot       : MainProcess                     : Thread-3       : INFO    : RUN ComputeUnit 'unit.000000' state changed from 'Scheduling' to 'AgentStagingInputPending'.
2015-09-18 16:41:56,838: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : Copying sdist 'file://localhost/home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/utils/radical.utils-v0.35-27-ged3c13d-devel.tar.gz' to sandbox (file://localhost/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/).
2015-09-18 16:41:56,944: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : Copying sdist 'file://localhost/home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/saga/saga-python-v0.35-14-g84133c5-devel.tar.gz' to sandbox (file://localhost/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/).
2015-09-18 16:41:57,050: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : Copying sdist 'file://localhost/home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/controller/..//radical.pilot-v0.35-381-ga9e567a-devel.tar.gz' to sandbox (file://localhost/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/).
2015-09-18 16:41:57,159: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : Writing agent configuration to file '/tmp/rp_agent_cfg_HHL7sv.json'.
2015-09-18 16:41:57,161: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : Copying agent configuration file 'file://localhost/tmp/rp_agent_cfg_HHL7sv.json' to sandbox (file://localhost/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/).
2015-09-18 16:41:57,368: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : saga.job.Service ('fork://localhost/')
2015-09-18 16:41:59,237: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : Bootstrap command line: /bin/bash ['-l bootstrap_1.sh', " -d 'radical.utils-v0.35-27-ged3c13d-devel.tar.gz:saga-python-v0.35-14-g84133c5-devel.tar.gz:radical.pilot-v0.35-381-ga9e567a-devel.tar.gz' -m 'create' -p 'pilot.0000' -r 'debug' -s 'rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024' -v '/home/vivek/radical.pilot.sandbox/ve_localhost' -a 'multicore'"]
2015-09-18 16:41:59,240: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : Submitting SAGA job with description: {'Project': None, 'Executable': '/bin/bash', 'TotalPhysicalMemory': None, 'WorkingDirectory': '/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/', 'Queue': None, 'Environment': {}, 'WallTimeLimit': 10, 'Arguments': ['-l bootstrap_1.sh', " -d 'radical.utils-v0.35-27-ged3c13d-devel.tar.gz:saga-python-v0.35-14-g84133c5-devel.tar.gz:radical.pilot-v0.35-381-ga9e567a-devel.tar.gz' -m 'create' -p 'pilot.0000' -r 'debug' -s 'rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024' -v '/home/vivek/radical.pilot.sandbox/ve_localhost' -a 'multicore'"], 'ProcessesPerHost': None, 'Error': 'bootstrap_1.err', 'Output': 'bootstrap_1.out', 'TotalCPUCount': 8}
2015-09-18 16:41:59,467: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : SAGA job submitted with job id [fork://localhost/]-[3989.0]
2015-09-18 16:42:00,053: radical.pilot       : MainProcess                     : Thread-1       : INFO    : ComputePilot 'pilot.0000' state changed from 'Launching' to 'PendingActive'.
[Callback]: ComputePilot 'pilot.0000' state: PendingActive.
2015-09-18 16:42:55,123: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: INFO    : Performing periodical health check for pilot.0000 (SAGA job id [fork://localhost/]-[3989.0])
2015-09-18 16:42:57,035: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: INFO    : pilot pilot.0000 seems alive and well
2015-09-18 16:43:02,262: radical.pilot       : MainProcess                     : Thread-1       : INFO    : ComputePilot 'pilot.0000' state changed from 'PendingActive' to 'Active'.
[Callback]: ComputePilot 'pilot.0000' state: Active.
2015-09-18 16:43:05,697: radical.pilot       : MainProcess                     : Thread-1       : INFO    : ComputePilot 'pilot.0000' state changed from 'Active' to 'Done'.
[Callback]: ComputePilot 'pilot.0000' state: Done.
2015-09-18 16:43:06,139: radical.pilot       : MainProcess                     : Thread-1       : INFO    : ComputePilot 'pilot.0000' state changed from 'Done' to 'Failed'.
[Callback]: ComputePilot 'pilot.0000' state: Failed.
2015-09-18 16:43:06,140: radical.pilot       : MainProcess                     : Thread-1       : ERROR   : pilot manager controller thread caught system exit -- forcing application shutdown
Traceback (most recent call last):
  File "/home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/controller/pilot_manager_controller.py", line 343, in run
    self.call_callbacks(pilot_id, new_state)
  File "/home/vivek/Research/ves/myenv/local/lib/python2.7/site-packages/radical/pilot/controller/pilot_manager_controller.py", line 238, in call_callbacks
    cb(self._shared_data[pilot_id]['facade_object'](), new_state)
  File "getting_started_local.py", line 30, in pilot_state_cb
    sys.exit (1)
SystemExit: 1

agent.err:

 cat agent.0.err 
Process Process-1:1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/rp_install/lib/python2.7/site-packages/radical/pilot/utils/queue.py", line 460, in _bridge
    _in.bind(addr_in)
  File "zmq/backend/cython/socket.pyx", line 489, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:4824)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7055)
    raise ZMQError(errno)
ZMQError: Address already in use
Process Process-1:2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/rp_install/lib/python2.7/site-packages/radical/pilot/utils/queue.py", line 460, in _bridge
    _in.bind(addr_in)
  File "zmq/backend/cython/socket.pyx", line 489, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:4824)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7055)
    raise ZMQError(errno)
ZMQError: Address already in use
Process Process-1:3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/rp_install/lib/python2.7/site-packages/radical/pilot/utils/queue.py", line 460, in _bridge
    _in.bind(addr_in)
  File "zmq/backend/cython/socket.pyx", line 489, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:4824)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7055)
    raise ZMQError(errno)
ZMQError: Address already in use
Process Process-1:4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/rp_install/lib/python2.7/site-packages/radical/pilot/utils/queue.py", line 460, in _bridge
    _in.bind(addr_in)
  File "zmq/backend/cython/socket.pyx", line 489, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:4824)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7055)
    raise ZMQError(errno)
ZMQError: Address already in use
Process Process-1:5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/rp_install/lib/python2.7/site-packages/radical/pilot/utils/pubsub.py", line 267, in _bridge
    _in.bind(addr_in)
  File "zmq/backend/cython/socket.pyx", line 489, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:4824)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7055)
    raise ZMQError(errno)
ZMQError: Address already in use
Process Process-1:6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/rp_install/lib/python2.7/site-packages/radical/pilot/utils/pubsub.py", line 267, in _bridge
    _in.bind(addr_in)
  File "zmq/backend/cython/socket.pyx", line 489, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:4824)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7055)
    raise ZMQError(errno)
ZMQError: Address already in use
Process Process-1:7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/rp_install/lib/python2.7/site-packages/radical/pilot/utils/pubsub.py", line 267, in _bridge
    _in.bind(addr_in)
  File "zmq/backend/cython/socket.pyx", line 489, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:4824)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7055)
    raise ZMQError(errno)
ZMQError: Address already in use
Process Process-1:8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vivek/radical.pilot.sandbox/rp.session.vivek-Lenovo-IdeaPad-Y480.vivek.016696.0024-pilot.0000/rp_install/lib/python2.7/site-packages/radical/pilot/utils/pubsub.py", line 267, in _bridge
    _in.bind(addr_in)
  File "zmq/backend/cython/socket.pyx", line 489, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:4824)
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7055)
    raise ZMQError(errno)
ZMQError: Address already in use
andre-merzky commented 8 years ago

Thanks for the report, vivek. It seems to be a termination issue in the pilot agent. Can you please check for lingering python processes (ps -ef | grep python) and kill them manually? I'll close this as a duplicate of #726. Thanks!