Closed Weiming-Hu closed 3 years ago
@Weiming-Hu , I wanted to tell you that I will be following this up along with the open ticket: https://github.com/radical-cybertools/radical.entk/issues/509, as this needs to be addressed in the use of pika. We anticipate this will be resolved no later than the January release.
Sounds good to me. I will be actively monitoring this thread. Thanks.
I have encountered this problem once. I'm using EnTK on NCAR Cheyenne. My radical stack info is as follows:
My workflow consists of two stages. I requested 6480 CPUs for 480 minutes (8 hours). However, when the second stage is complete (roughly after 2 hours), EnTK did not terminate as expected, instead it
EnTK was stuck at the
submitting
stage but the tasks were complete with all the output ready. At a certain point, when I went back on Cheyenne to check the progress, I found that my data were already generated but EnTK was still running. When Issh
into one of the compute nodes, I notice severalrp.*
processes were sleeping. Please see the following output fromtop
on one of the compute nodes when EnTK was stuck.I then terminated EnTK manually.
There are no error messages in the client sandbox when I run
grep -R Error *
; there are no error messages either in the server sandbox when I rungrep -R Error *.log
.I'm not sure whether this issue is related to another connection issue I opened at EnTK. This is my second time finishing an entire workflow. For the first time, I didn't have this problem though.
Thank you all for your help.