Closed euhruska closed 5 years ago
2019-02-06 15:19:36,377: radical.entk.wfprocessor.0000: MainProcess : MainThread : DEBUG : Attempting to end WFprocessor... event: False 2019-02-06 15:19:36,424: radical.entk.wfprocessor.0000: wfprocessor : MainThread : INFO : WFprocessor started 2019-02-06 15:19:36,424: radical.entk.wfprocessor.0000: wfprocessor : MainThread : INFO : Terminating enqueue-thread 2019-02-06 15:19:36,424: radical.entk.wfprocessor.0000: wfprocessor : MainThread : ERROR : Error in wfp process: 'NoneType' object has no attribute 'join'. Closing enqueue, dequeue threads 2019-02-06 15:19:36,424: radical.entk.wfprocessor.0000: wfprocessor : MainThread : INFO : WFprocessor process terminated Process wfprocessor: Traceback (most recent call last): File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap self.run() File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/site-packages/radical/entk/appman/wfprocessor.py", line 557, in _wfp raise EnTKError(text=ex) TypeError: __init__() got an unexpected keyword argument 'text' Traceback (most recent call last): File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/site-packages/radical/entk/appman/wfprocessor.py", line 503, in _wfp self._enqueue_thread.join() AttributeError: 'NoneType' object has no attribute 'join' 2019-02-06 15:19:36,431: radical.entk.wfprocessor.0000: MainProcess : MainThread : DEBUG : WFprocessor process terminated 2019-02-06 15:19:36,433: radical.entk.appmanager.0000: MainProcess : MainThread : INFO : Terminating synchronizer thread 2019-02-06 15:19:37,049: radical.entk.appmanager.0000: MainProcess : MainThread : INFO : Synchronizer thread terminated 2019-02-06 15:19:46,295: radical.entk.resource_manager.0000: MainProcess : pmgr.0000.idler._state_pull_cb: INFO : Pilot pilot.0000 state: CANCELED 2019-02-06 15:19:52,445: radical.utils : MainProcess : MainThread : DEBUG : lm create pool for gsisftp://bw.ncsa.illinois.edu/shell_file_adaptor_command_shell/ (<type 'str'>) (<radical.utils.lease_manager.LeaseManager object at 0x7f4d589dbf10>) 2019-02-06 15:19:52,445: radical.utils : MainProcess : MainThread : DEBUG : lm create object for gsisftp://bw.ncsa.illinois.edu/shell_file_adaptor_command_shell/ Error: [Errno 5] Input/output error Traceback (most recent call last): File "extasy_tica3.py", line 322, in <module> appman.run() File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/site-packages/radical/entk/appman/appmanager.py", line 344, in run port=self._port File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/site-packages/radical/entk/execman/rp/task_manager.py", line 51, in __init__ rts='radical.pilot') File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/site-packages/radical/entk/execman/base/task_manager.py", line 79, in __init__ self._uid = ru.generate_id('task_manager.%(item_counter)04d', ru.ID_CUSTOM, namespace=self._sid) File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/site-packages/radical/utils/ids.py", line 179, in generate_id return _generate_id(template, prefix, namespace) File "/scratch1/eh22/conda/envs/extasy16/lib/python2.7/site-packages/radical/utils/ids.py", line 248, in _generate_id os.close(fd) OSError: [Errno 5] Input/output error
Hey Eugene, can you add the entire verbose log? The message from the event causing the error might be missing.
looks like this was caused by the bluewaters storage maintenance, now fixed