search

SmileiPIC / Smilei

Particle-in-cell code for plasma simulation
https://smileipic.github.io/Smilei
335 stars 119 forks source link

Crash with GPU computing #685

Closed Tissot11 closed 8 months ago

Tissot11 commented 8 months ago

Hi,

I managed to compiled the Smilei for Nvidia GPUs. But when I am running simulations, they are crashing with this error

ib_md.c:1550 UCX WARN mlx5_1: relaxed order memory access requested, but unsupported

What could be the reason.

I used following modules

module load nvhpcsdk/23 cuda/11.8-nvhpcsdk openmpi/4.1 openmpi_gpu/4.1 hdf5-mpi/1.12.2 anaconda/3/2023.03

What could be the reason for the crash?

Does Smilei only support OpenMP for AMD GPUs and not Nvidia?

Tissot11 commented 8 months ago

I attach the std err and out files. According to the admins, the warning is not the cause of crash. They want to know where exactly the crash happens.

tjob_hybrid.err.8969072.txt tjob_hybrid.out.8969072.txt

mccoys commented 8 months ago

It looks like you are using the injector, but I am not sure the injector has been ported to GPU yet. @charlesprouveur ?

Tissot11 commented 8 months ago

Yes, I'm using injectors. If this is not ported yet then perhaps I don't need to use GPUs for the time being.

Tissot11 commented 8 months ago

If Particle injectors being not ported to GPU was the reason for the crash, then you can close this ticket. When do you expect to support particle injectors on GPU?

beck-llr commented 8 months ago

Particle injector has not been ported yet on GPU and we can not tell when it will be. I can tell that unfortunately it is not very high on our priority list at the moment because there are so many other things to do. Any help on the topic is welcome by the way.