Closed wjlei1990 closed 3 years ago
Another error run:
EnTK session: re.session.login5.lei.018844.0010
Creating AppManagerSetting up RabbitMQ system ok
ok
Validating and assigning resource manager ok
Setting up RabbitMQ system n/a
new session: [re.session.login5.lei.018844.0010] \
database : [mongodb://hpcw-pr:****@129.114.17.185:27017/hpcw-pr] ok
create pilot manager ok
submit 1 pilot(s)
pilot.0000 ornl.summit 215040 cores 7680 gpus ok
closing session re.session.login5.lei.018844.0010 \
close pilot manager \
wait for 1 pilot(s)
0 ok
ok
session lifetime: 59.4s ok
wait for 1 pilot(s)
0 timeout
Execution failed, error: 'NoneType' object has no attribute '_uid'
Traceback (most recent call last):
File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/entk/appman/appmanager.py", line 438, in run
self._rmgr.submit_resource_request()
File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/entk/execman/rp/resource_manager.py", line 199, in submit_resource_request
self._pilot.wait([rp.PMGR_ACTIVE, rp.DONE, rp.FAILED, rp.CANCELED])
File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/pilot/pilot.py", line 558, in wait
time.sleep(0.1)
KeyboardInterrupt
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_entk.py", line 103, in main
appman.run()
File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/entk/appman/appmanager.py", line 462, in run
self.terminate()
File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/entk/appman/appmanager.py", line 507, in terminate
write_session_description(self)
File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/entk/utils/prof_utils.py", line 148, in write_session_description
tree[amgr._uid]['children'].append(wfp._uid)
AttributeError: 'NoneType' object has no attribute '_uid'
radical-stack
python : /ccs/home/lei/.conda/envs/summit-entk/bin/python3
pythonpath : /sw/summit/xalt/1.2.1/site:/sw/summit/xalt/1.2.1/libexec
version : 3.7.6
virtualenv : summit-entk
radical.analytics : 1.6.7
radical.entk : 1.6.7
radical.gtod : 1.6.7
radical.pilot : 1.6.7
radical.saga : 1.6.10
radical.utils : 1.6.7
This issues seems to be related to my .bashrc
file.
I had a very lengthy bashrc that takes a while to load on Summit. I think today Summit is kind of slow so it tooke longer than before to load the bashrc file.
After cleaning the bashrc file a bit, I can get most of the jobs submitted.
Thanks for the help.
Let me know if you have updates in the future.
@wjlei1990 - you could try to set this env variable on the client side, before running your script:
export RADICAL_SAGA_PTY_SSH_TIMEOUT=60
the default timeout is 10 seconds, which may be indeed too short if your bash startup takes too long.
Hi I constantly encountered this issue when using entk on summit.
This issue doesn't not always happen but sometime it just pops out...I am not sure what is the reason...Could you help us to figure them out?