search

CompPhysVienna / n2p2

n2p2 - A Neural Network Potential Package
https://compphysvienna.github.io/n2p2/
GNU General Public License v3.0
214 stars 81 forks source link

Excessive time spent on MPI Bcast in nnp-train #177

Open mikejwaters opened 1 year ago

mikejwaters commented 1 year ago

I am running n2p2 on NERSC's newest, machine Perlmutter (https://docs.nersc.gov/performance/readiness/) using their PrgEnv-gnu compiler wrappers (https://docs.nersc.gov/systems/perlmutter/software/#compilers). After seeing terrible performance, the NERSC staff and I profiled the performance and found 98% of the time was spent on MPI with 96.4% of time spent on MPI_Bcast.

While the NERSC staff continues to work their vendors software team, is there anything I can try to mitigate this?

Edit: I'm on the master branch.

singraber commented 1 year ago

Hello!

What kind of run did you execute? Was it nnp-train or a LAMMPS/n2p2 run? What kind of nodes and how many cores did you use? Please provide as much information as you have otherwise I can only speculate...

Best, Andreas