search

pelahi / VELOCIraptor-STF

Galaxy/(sub)Halo finder for N-body simulations
MIT License
19 stars 26 forks source link

[Performance bug]: Spherical over density with OpenMP #29

Closed JBorrow closed 5 years ago

JBorrow commented 5 years ago

It appears that the spherical over density search is very slow when running in OMP only mode -- I think it's only single-threaded. It takes over 8 hours (when my job was cancelled) running on 16 threads to run on a 512^3 box.

Also -- I've noticed that the 3D FoF is not OMP parallel; is this coming in the future?

Is the current suggestion to run in MPI mode with 1 rank per core?

pelahi commented 5 years ago

Hi Josh, The master branch has a OpenMP 3DFoF now. I need to update the openmp routines to full enable it for the extra baryon searches but shouldn't take too long. As for the bug, I am not certain what the issue could be. There was a bug in the branch that is now fixed but it shouldn't have caused a slow down as the code was always OpenMP. I've merged the latest changes into the extrabaryon branch. Can you please try the search again and let me know if there continues to be an issue? Also, are you calculating spherical overdensities when searching for stars +(gas+bh)? May I ask why? SO masses, unless at higher overdensities than 200, are only really meaningful if you include dark matter.

JBorrow commented 5 years ago

Great, I'll try again with the latest master. Thank you.

As for the SO searches, I was hoping to run them basically on the DM only, yes -- should I be running VR in DM only mode for these to be valid?

JBorrow commented 5 years ago

The OMP parallel 3D FoF is great, thank you! As for the spherical over-density, even with the latest master it seems that my code hangs, taking at least 48 hours (before I killed the job) to do the SO search on a 512^3 box. I'm happy to provide the files if you require them. I must be doing something wrong 🤷‍♂️. I initially thought that my G was in the incorrect units, but even after fixing that things seem to still hang.

FWIW this is using the SIMBA interface (4); perhaps it's something unique to this? Running SO on my SWIFT snapshots does not result in this problem.

pelahi commented 5 years ago

Hi Josh, can you give me more information regarding the SIMBA run?