search

radical-cybertools / ExTASY

MDEnsemble
Other
1 stars 1 forks source link

SAGA(?) Could not detect shell prompt on ARCHER #235

Closed ibethune closed 8 years ago

ibethune commented 8 years ago

If I try running one of the example workflows (here coco-amber) I get the following error:

RADICAL_PILOT_DEBUG=Verbose python extasy_amber_coco.py --RPconfig archer.rcfg --Kconfig cocoamber.wcfg

================================================================================
 EnsembleMD (0.3.14)                                                            
================================================================================

Starting Allocation2016-01-21 09:21:10,930: radical.saga.pty    : MainProcess                     : MainThread     : ERROR   : Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)"))
Traceback (most recent call last):
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 288, in _initialize_pty
    raise se.NoSuccess ("Could not detect shell prompt (timeout)")
NoSuccess: Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)"))
2016-01-21 09:21:10,930: radical.enmd.SingleClusterEnvironment: MainProcess                     : MainThread     : ERROR   : Fatal error during resource allocation: Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)")).
Traceback (most recent call last):
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/ensemblemd/single_cluster_environment.py", line 181, in allocate
    self._pilot = pmgr.submit_pilots(pdesc)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/pilot_manager.py", line 348, in submit_pilots
    resource_config=resource_cfg)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/controller/pilot_manager_controller.py", line 426, in register_start_pilot_request
    shell = sup.PTYShell(url, self._session)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell.py", line 243, in __init__
    posix=self.posix)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 196, in initialize
    self._initialize_pty (info['pty'], info)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 411, in _initialize_pty
    raise ptye.translate_exception (e)
NoSuccess: Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)"))
Allocation failed: Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)"))Traceback (most recent call last):
  File "extasy_amber_coco.py", line 197, in <module>
    cluster.allocate()
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/ensemblemd/single_cluster_environment.py", line 181, in allocate
    self._pilot = pmgr.submit_pilots(pdesc)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/pilot_manager.py", line 348, in submit_pilots
    resource_config=resource_cfg)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/controller/pilot_manager_controller.py", line 426, in register_start_pilot_request
    shell = sup.PTYShell(url, self._session)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell.py", line 243, in __init__
    posix=self.posix)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 196, in initialize
    self._initialize_pty (info['pty'], info)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 411, in _initialize_pty
    raise ptye.translate_exception (e)
saga.exceptions.NoSuccess: Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)"))
(extasy-test3)wlandhcp165:coam-on-archer ibethune$ RADICAL_PILOT_VERBOSE=DEbug python extasy_amber_coco.py --RPconfig archer.rcfg --Kconfig cocoamber.wcfg
2016-01-21 09:21:47,176: radical.pilot       : MainProcess                     : MainThread     : INFO    : python.interpreter   version: 2.7.11 (default, Jan 15 2016, 09:52:45) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.72)]
2016-01-21 09:21:47,176: radical.pilot       : MainProcess                     : MainThread     : INFO    :                      pid: 9945
2016-01-21 09:21:47,176: radical.pilot       : MainProcess                     : MainThread     : INFO    :                      tid: MainThread
2016-01-21 09:21:47,176: radical.pilot       : MainProcess                     : MainThread     : INFO    : radical.pilot        version: 0.38

================================================================================
 EnsembleMD (0.3.14)                                                            
================================================================================

Starting Allocation2016-01-21 09:21:47,505: radical.pilot       : MainProcess                     : MainThread     : INFO    : using database mongodb://extasy:extasyproject@extasy-db.epcc.ed.ac.uk/radicalpilot
2016-01-21 09:21:47,610: radical.pilot       : MainProcess                     : MainThread     : INFO    : New Session created: {'database_url': 'mongodb://extasy:extasyproject@extasy-db.epcc.ed.ac.uk/radicalpilot', 'connected': 1453368107.567685, 'uid': 'rp.session.wlandhcp165.dl.ac.uk.ibethune.016821.0002', 'closed': None, 'created': 1453368107.567685}.
2016-01-21 09:21:47,611: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_aliases.json
2016-01-21 09:21:47,611: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_das4.json
2016-01-21 09:21:47,616: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for das4.fs2
2016-01-21 09:21:47,617: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_epsrc.json
2016-01-21 09:21:47,623: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for epsrc.archer
2016-01-21 09:21:47,624: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for epsrc.archer_orte
2016-01-21 09:21:47,625: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_futuregrid.json
2016-01-21 09:21:47,642: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.xray
2016-01-21 09:21:47,643: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.echo
2016-01-21 09:21:47,644: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.india
2016-01-21 09:21:47,645: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.bravo
2016-01-21 09:21:47,646: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.xray_ccm
2016-01-21 09:21:47,647: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for futuregrid.delta
2016-01-21 09:21:47,648: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_iu.json
2016-01-21 09:21:47,656: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for iu.bigred2
2016-01-21 09:21:47,657: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for iu.bigred2_ccm
2016-01-21 09:21:47,658: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_local.json
2016-01-21 09:21:47,660: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for local.localhost
2016-01-21 09:21:47,661: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_lrz.json
2016-01-21 09:21:47,665: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for lrz.supermuc
2016-01-21 09:21:47,666: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_ncar.json
2016-01-21 09:21:47,669: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for ncar.yellowstone
2016-01-21 09:21:47,670: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_ncsa.json
2016-01-21 09:21:47,679: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for ncsa.bw_ccm
2016-01-21 09:21:47,680: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for ncsa.bw_aprun
2016-01-21 09:21:47,681: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for ncsa.bw
2016-01-21 09:21:47,682: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_nersc.json
2016-01-21 09:21:47,708: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.edison
2016-01-21 09:21:47,709: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.edison_aprun
2016-01-21 09:21:47,711: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.hopper_ccm
2016-01-21 09:21:47,712: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.hopper_aprun
2016-01-21 09:21:47,713: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.edison_ccm
2016-01-21 09:21:47,714: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for nersc.hopper
2016-01-21 09:21:47,715: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_ornl.json
2016-01-21 09:21:47,719: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for ornl.titan
2016-01-21 09:21:47,720: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_radical.json
2016-01-21 09:21:47,723: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for radical.tutorial
2016-01-21 09:21:47,724: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_rice.json
2016-01-21 09:21:47,730: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for rice.biou
2016-01-21 09:21:47,731: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for rice.davinci
2016-01-21 09:21:47,732: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_stfc.json
2016-01-21 09:21:47,735: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for stfc.joule
2016-01-21 09:21:47,736: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations from /Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/configs/resource_xsede.json
2016-01-21 09:21:47,758: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.blacklight
2016-01-21 09:21:47,759: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.supermic
2016-01-21 09:21:47,760: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.lonestar
2016-01-21 09:21:47,761: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.comet
2016-01-21 09:21:47,762: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.stampede
2016-01-21 09:21:47,763: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.trestles
2016-01-21 09:21:47,764: radical.pilot       : MainProcess                     : MainThread     : INFO    : Load resource configurations for xsede.gordon
2016-01-21 09:21:47,790: radical.pilot       : MainProcess                     : Thread-1       : DEBUG   : Worker thread (ID: Thread-1[123145310715904]) for PilotManager pmgr.0000 started.
2016-01-21 09:21:47,795: radical.pilot       : MainProcess                     : PilotLauncherWorker-1: DEBUG   : Connected to MongoDB. Serving requests for PilotManager pmgr.0000.
2016-01-21 09:21:47,798: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : saga.utils.PTYShell ('ssh://login.archer.ac.uk/')
2016-01-21 09:21:58,931: radical.saga.pty    : MainProcess                     : MainThread     : ERROR   : Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)"))
Traceback (most recent call last):
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 288, in _initialize_pty
    raise se.NoSuccess ("Could not detect shell prompt (timeout)")
NoSuccess: Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)"))
2016-01-21 09:21:58,931: radical.enmd.SingleClusterEnvironment: MainProcess                     : MainThread     : ERROR   : Fatal error during resource allocation: Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)")).
Traceback (most recent call last):
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/ensemblemd/single_cluster_environment.py", line 181, in allocate
    self._pilot = pmgr.submit_pilots(pdesc)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/pilot_manager.py", line 348, in submit_pilots
    resource_config=resource_cfg)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/controller/pilot_manager_controller.py", line 426, in register_start_pilot_request
    shell = sup.PTYShell(url, self._session)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell.py", line 243, in __init__
    posix=self.posix)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 196, in initialize
    self._initialize_pty (info['pty'], info)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 411, in _initialize_pty
    raise ptye.translate_exception (e)
NoSuccess: Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)"))
Allocation failed: Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)"))2016-01-21 09:21:58,932: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : session rp.session.wlandhcp165.dl.ac.uk.ibethune.016821.0002 closing
2016-01-21 09:21:58,932: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : session rp.session.wlandhcp165.dl.ac.uk.ibethune.016821.0002 closes   pmgr   pmgr.0000
2016-01-21 09:21:58,932: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pmgr    pmgr.0000 closing
2016-01-21 09:21:58,932: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pmgr    pmgr.0000 cancel   launcher Thread-1
2016-01-21 09:21:58,932: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pworker Thread-1 disables launcher PilotLauncherWorker-1
2016-01-21 09:21:58,932: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : launcher PilotLauncherWorker-1 disabling
2016-01-21 09:21:58,932: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : launcher PilotLauncherWorker-1 disabled
2016-01-21 09:21:58,932: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pworker Thread-1 disabled launcher PilotLauncherWorker-1
2016-01-21 09:21:58,932: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pmgr    pmgr.0000 canceled launcher Thread-1
2016-01-21 09:21:58,943: radical.pilot       : MainProcess                     : MainThread     : INFO    : Sent 'COMMAND_CANCEL_PILOT' command to pilots [].
2016-01-21 09:21:58,943: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pmgr    pmgr.0000 cancel   worker Thread-1
2016-01-21 09:21:58,943: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pworker Thread-1 stops   launcher PilotLauncherWorker-1
2016-01-21 09:21:58,943: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : launcher PilotLauncherWorker-1 stopping
2016-01-21 09:21:58,982: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : launcher PilotLauncherWorker-1 stopped
2016-01-21 09:21:58,982: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pworker Thread-1 stopped launcher PilotLauncherWorker-1
2016-01-21 09:21:58,982: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pmgr    pmgr.0000 canceled worker Thread-1
2016-01-21 09:21:58,982: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pmgr    pmgr.0000 stops    worker Thread-1
2016-01-21 09:21:58,982: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pworker Thread-1 stopping
2016-01-21 09:21:59,958: radical.pilot       : MainProcess                     : Thread-1       : DEBUG   : pworker Thread-1 stops   launcher PilotLauncherWorker-1
2016-01-21 09:21:59,958: radical.pilot       : MainProcess                     : Thread-1       : DEBUG   : launcher PilotLauncherWorker-1 stopping
2016-01-21 09:21:59,958: radical.pilot       : MainProcess                     : Thread-1       : DEBUG   : launcher PilotLauncherWorker-1 stopped
2016-01-21 09:21:59,958: radical.pilot       : MainProcess                     : Thread-1       : DEBUG   : pworker Thread-1 stopped launcher PilotLauncherWorker-1
2016-01-21 09:21:59,958: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pworker Thread-1 stopped
2016-01-21 09:21:59,958: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pmgr    pmgr.0000 stopped  worker Thread-1
2016-01-21 09:21:59,958: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : pmgr    pmgr.0000 closed
2016-01-21 09:21:59,959: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : session rp.session.wlandhcp165.dl.ac.uk.ibethune.016821.0002 closed   pmgr   pmgr.0000
2016-01-21 09:22:00,051: radical.pilot       : MainProcess                     : MainThread     : DEBUG   : session rp.session.wlandhcp165.dl.ac.uk.ibethune.016821.0002 closed
Traceback (most recent call last):
  File "extasy_amber_coco.py", line 197, in <module>
    cluster.allocate()
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/ensemblemd/single_cluster_environment.py", line 181, in allocate
    self._pilot = pmgr.submit_pilots(pdesc)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/pilot_manager.py", line 348, in submit_pilots
    resource_config=resource_cfg)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/radical/pilot/controller/pilot_manager_controller.py", line 426, in register_start_pilot_request
    shell = sup.PTYShell(url, self._session)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell.py", line 243, in __init__
    posix=self.posix)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 196, in initialize
    self._initialize_pty (info['pty'], info)
  File "/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py", line 411, in _initialize_pty
    raise ptye.translate_exception (e)
saga.exceptions.NoSuccess: Could not detect shell prompt (timeout) (/Users/ibethune/Desktop/extasy-test3/lib/python2.7/site-packages/saga/utils/pty_shell_factory.py +288 (_initialize_pty)  :  raise se.NoSuccess ("Could not detect shell prompt (timeout)"))

I have the following versions installed (basically everything from the latest releases on pip):

(extasy-test3)wlandhcp165:coam-on-archer ibethune$ radicalpilot-version 
0.38
(extasy-test3)wlandhcp165:coam-on-archer ibethune$ ensemblemd-version 
0.3.14
(extasy-test3)wlandhcp165:coam-on-archer ibethune$ radical-utils-version 
0.38
(extasy-test3)wlandhcp165:coam-on-archer ibethune$ sagapython-version 
0.40

The only thing I know which might be related to this is that due to a recent security issue in OpenSSH (https://access.redhat.com/articles/2123781) ARCHER will now reject all incoming SSH connections which have the 'roaming' feature enabled in the client. The manual fix is to edit the .ssh/config:

echo "UseRoaming no" >> ~/.ssh/config

I don't know if this is the root cause of the problem or not, but worth investigating!

andre-merzky commented 8 years ago

Hi Iain,

does the use of echo "UseRoaming no" >> ~/.ssh/config make any difference then?

Thanks, Andre.

ibethune commented 8 years ago

No, I have that in my .ssh/config already (so I can log into ARCHER manually using SSH), and it was set when I hit the error above. I just wondering if SAGA/RP was doing something different internally. It might well be a red-herring.

ashkurti commented 8 years ago

I have the same problem as Iain does now with output as in https://gist.github.com/ashkurti/30fae1a847a8ef05414d

marksantcroos commented 8 years ago

Can you try to run touch .hushlogin once logged into archer?

andre-merzky commented 8 years ago

Hmm, bugger, I am not able to reproduce it: I can run the RP archer test just fine, w/o any change of ssh settings...

Iain, if you don't mind, would you please run another test with SAGA_VERBOSE=DEBUG, and post the (longish) output somewhere? Thanks!

ashkurti commented 8 years ago

ardi@eslogin001:~> touch .hushlogin ardi@eslogin001:~>

marksantcroos commented 8 years ago

Andre: does saga use -q? Those two together should turn off all verbosity.

andre-merzky commented 8 years ago

@marksantcroos: I don't have hushlogin set... :/

marksantcroos commented 8 years ago

@andre-merzky thats because you are behind a dialup link

andre-merzky commented 8 years ago

@marksantcroos: no, no -q used...

What would be the difference of the dialup?

marksantcroos commented 8 years ago

@ashkurti And then retry again.

marksantcroos commented 8 years ago

@marksantcroos: no, no -q used...

Why not? :)

marksantcroos commented 8 years ago

My archer ssh config entry looks like:

host archer login.archer.ac.uk
    hostname login.archer.ac.uk
    user marksant
    LogLevel QUIET

With this and hushlogin I can get rid of most prompt detection errors.

ibethune commented 8 years ago

Gah, now it's working... If Ardita can confim it's now working as well, we can close. Some temporary glitch...

andre-merzky commented 8 years ago

@marksantcroos: no, no -q used... Why not? :)

Let's focus ;)

ashkurti commented 8 years ago

My job is in the queue and before running it again I did launch from ARCHER the "touch .hushlogin" command.

marksantcroos commented 8 years ago

Let's focus ;)

I dont understand that comment.

marksantcroos commented 8 years ago

Gah, now it's working... If Ardita can confim it's now working as well, we can close. Some temporary glitch...

Its a timing issue, likely not a glitch.

ibethune commented 8 years ago

So you still want to see a recreate with SAGA_VERBOSE=Debug ?

andre-merzky commented 8 years ago

@ibethune : For a failing run, yes, please!

@marksantcroos : -q might make sense, but it should also work without, so I'd rather first try to find why it hangs before cleaning up with -q... Am I missing something?

ashkurti commented 8 years ago

@ibethune - So did it work fine without any errors similar to the ones that @vivek-bala was having yesterday?

ibethune commented 8 years ago

No I get problems with coco. I'm going to file a separate ticket for that. It's to do with which modules are loaded.

This one will be closed later if I don't get a recreate of the original problem.

ashkurti commented 8 years ago

My job is still in the queue ... I do expect it to behave like yours though, not to have any problems with the shell but to have problems with the coco-installation. I tried running the coco workflow on stampede that also uses the 0-25 version and everything works just fine on stampede.

ashkurti commented 8 years ago

I did not get any problems with the shell access now on ARCHER, but more at a coco installation level as explained in Issue #236

I think we could close this issue unless somebody is willing to investigate the weird behaviour happening with ARCHER for the shell login access.

ibethune commented 8 years ago

Closing, unless it re-occurs