Very long allocating memory or initializating by random_symmetric

[rodionstepanov]

Hi! I run RBC convection in a box 1024x512x512. After message 'Info: Allocating memory. ..' to run simulation it takes around 5 mins for Tesla V100 on Linux cluster but for the same code for RTX-3090Ti on Windows PC it takes a few seconds. 1024x1024x512 is 10 mins and 1024^3 is 20 min. Experimentally I've found that it delays caused by a call of random_symmetric function.

[ProjectPhysX]

Hi @rodionstepanov,

interesting observation, I wasn't aware of this. random_symmetric() is no more than a std::rand() call internally. During initialization you should see the CPU go to full load on all cores, on both systems; I added this multithreading in update 2.9. Maybe the CPU in your Linux system is weaker? Or maybe the rand() call with g++ does something weird? I'll test this myself and try to replicate.

Thanks and kind regards, Moritz

[ProjectPhysX]

Hi @rodionstepanov,

I can replicate this issue, full 8 minutes for 1024x512x512 on 2x Xeon E5-2680 v3, 48 threads. Digging in the internet, I found the root cause: gcc's std::rand() is dog slow with multithreading, as it contains a mutex lock.

If I change the initialization loop in main_setup() from

parallel_for(lbm.get_N(), [&](ulong n) { ... });

to

for(ulong n=0ull; n<lbm.get_N(); n++) { ... }

to run single-threaded, it's down to 2 minutes from 8.

I have now replaced std::rand() with the standard C99 LCG function; this is suitable for multithreading and brings multithreaded initialization down to 1 minute. Please update!

Kind regards, Moritz

[rodionstepanov]

@ProjectPhysX , thanks a lot. I've updated and it is fast and furious now 🥇