can't start new thread

[euhruska]

why did this happen?

Process task-manager:
Traceback (most recent call last):
  File "/scratch1/eh22/conda/envs/extasy11/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/scratch1/eh22/conda/envs/extasy11/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/scratch1/eh22/conda/envs/extasy11/lib/python2.7/site-packages/radical/entk/execman/rp/task_manager.py", line 271, in _tmgr
    raise EnTKError(text=ex)
TypeError: __init__() got an unexpected keyword argument 'text'
Traceback (most recent call last):
  File "/scratch1/eh22/conda/envs/extasy11/lib/python2.7/site-packages/radical/entk/execman/rp/task_manager.py", line 177, in _tmgr
    umgr = rp.UnitManager(session=rmgr._session)
  File "/scratch1/eh22/conda/envs/extasy11/lib/python2.7/site-packages/radical/pilot/unit_manager.py", line 118, in __init__
    self.start(spawn=False)
  File "/scratch1/eh22/conda/envs/extasy11/lib/python2.7/site-packages/radical/utils/process.py", line 516, in start
    self._ru_initialize()
  File "/scratch1/eh22/conda/envs/extasy11/lib/python2.7/site-packages/radical/utils/process.py", line 837, in _ru_initialize
    self.ru_initialize_common()
  File "/scratch1/eh22/conda/envs/extasy11/lib/python2.7/site-packages/radical/pilot/utils/component.py", line 551, in ru_initialize_common
    self._log)
  File "/scratch1/eh22/conda/envs/extasy11/lib/python2.7/site-packages/radical/pilot/utils/component.py", line 252, in start_components
    comp.start()
  File "/scratch1/eh22/conda/envs/extasy11/lib/python2.7/site-packages/radical/utils/process.py", line 510, in start
    (self._ru_name, msg, timeout))
RuntimeError: umgr.0000.staging.output.0 got unexpected message (error: error("can't start new thread",)) [20.0]

[vivek-bala]

Can you attach the entire log and the log files? Hard to tell without them

[euhruska]

entk-chignolin2-ref.log rp.session.leonardo.rice.edu.eh22.017765.0032.zip

[andre-merzky]

I would think we hit a threading limit. Is this a new (or updated) machine you are using here?

[euhruska]

no, save machine and settings as before, but I suspect that all the past not fully closed runs were taking up threads and that accumulated. A full restart of whole tmux environment fixed it