Tracking ENTK Performance Test on Summit

[wjlei1990]

I have conducted a few test cases on Summit using radical.entk.

Each task is using 384 GPU (64 nodes), which is the same size as our current production case.

The current tests contains slurm jobs that can run simulataneous tasks 1, 5, 10, and 20 at the same time. Those slurm jobs used 64, 320, 640 and 1280 nodes on Summit.

We will further analyze the results using radical.analytics.

If I want to use radical.analytics to get some performance metrics, is radical.analytics online doc a good starting point?

[wjlei1990]

I noticed on summit, if entk failed to launch the job, then it pops the exception:

EnTK session: re.session.login1.lei.018775.0006
Creating AppManagerSetting up RabbitMQ system                                 ok
                                                                              ok
Validating and assigning resource manager                                     ok
Setting up RabbitMQ system                                                   n/a
new session: [re.session.login1.lei.018775.0006]                               \
database   : [mongodb://hpcw-pr:****@129.114.17.185:27017/hpcw-pr]            ok
create pilot manager                                                          ok
submit 1 pilot(s)
        pilot.0000   ornl.summit           53760 cores    1920 gpus           ok
closing session re.session.login1.lei.018775.0006                              \
close pilot manager                                                            \
wait for 1 pilot(s)
              0                                                               ok
                                                                              ok
session lifetime: 38.5s                                                       ok
wait for 1 pilot(s)
              0                                                          timeout
Execution failed, error: 'NoneType' object has no attribute '_uid'
Traceback (most recent call last):
  File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/entk/appman/appmanager.py", line 433, in run
    self._rmgr.submit_resource_request()
  File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/entk/execman/rp/resource_manager.py", line 199, in submit_resource_request
    self._pilot.wait([rp.PMGR_ACTIVE, rp.DONE, rp.FAILED, rp.CANCELED])
  File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/pilot/pilot.py", line 558, in wait
    time.sleep(0.1)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_entk.hrlee.py", line 184, in main
    appman.run()
  File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/entk/appman/appmanager.py", line 457, in run
    self.terminate()
  File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/entk/appman/appmanager.py", line 502, in terminate
    write_session_description(self)
  File "/ccs/home/lei/.conda/envs/summit-entk/lib/python3.7/site-packages/radical/entk/utils/prof_utils.py", line 148, in write_session_description
    tree[amgr._uid]['children'].append(wfp._uid)
AttributeError: 'NoneType' object has no attribute '_uid'

It is not a big deal. I think it may be just a small bug (that raised an unhandled exception) in the entk software.

[wjlei1990]

Hi @mturilli, I am wondering if there is any instructions for the performance analysis?

[mturilli]

Hi @wjlei1990, happy to try to run the same analysis we used for the previous proposal on your new session. Let me know where I can find it and I will see what we can get out of it.

[wjlei1990]

Hi @mturilli, thanks for the help. The client log is located here:

/gpfs/alpine/world-shared/geo111/lei/entk

Then sandbox log is located here:

/gpfs/alpine/world-shared/geo111/lei/entk/sandbox

[mturilli]

@wjlei1990 , please find the update notebook for INCITE 2021 at: https://github.com/radical-experiments/hpcw-princeton/blob/master/analysis/incite2021.ipynb

I used the latest session at the specified path as it seemed the only one that had successfully completed (re.session.login1.lei.018775.0008)

[wjlei1990]

@mturilli it is weird...because what i saw on the terminal is that all the jobs are done. I checked the ouptut files they are good...

What do you suggest me to do? Maybe I can submit all the jobs one more time.

[mturilli]

Thanks @wjlei1990. I double checked and found all the profiles I needed. I pushed an iteration of the notebook with the plots of all the sessions.

[wjlei1990]

@mturilli Cool! Thanks very much. I think we can discusss it a bit more during our next meeting, and what is the next thing to be done for our paper!

[wjlei1990]

Hi @mturilli, I did another performance measurement on Summit using real simulation cases.

The thrid case is not so perfect. I think there are some slow nodes involved in the simulation.

So I am going to relaunch this one and update the figure.