search

ubermag / help

Repository for raising issues and requesting help on Ubermag
BSD 2-Clause "Simplified" License
11 stars 3 forks source link

Ubermag Parallel Computing Type #194

Closed MichaelSherburne closed 2 years ago

MichaelSherburne commented 2 years ago

I'm looking at using Ubermag at a cluster and would like to know what type of parallel computation Ubermag uses when it uses more cores?

Which of the following does it use below or is it a different parallel computation all together? -MPI (true parallel) -Multi-threaded (OpenMP) -Hybrid (MPI/OpenMP) -Many instances of an identical code run on individual nodes with each node using different input data?

Motivation for moving to a cluster has been in trying to run a room temperature simulation of a nanocrystal. I tried going to 70 unit cells at 1e-14s time steps but increasing the number of unit cells does not help convergence in this case. In addition, a larger number of unit cells creates more very high frequency components that are extraneous. I was recommended to try out a single unit cell by a professor and keep lowering the time step. This seems to start making the output look more like the data from VAMPIRE in a recent Nature article. However, the time response keeps changing as I lower the time step (1e-15s is still not enough). I think it may need to go down towards 1e-17s to remain consistent.

fangohr commented 2 years ago

Dear @MichaelSherburne,

if you use the OOMMF computational backend of Ubermag, you are restricted to the parallelisation that OOMMF offers. This is done through threading within a shared-memory machine (I think), so that you can use multiple CPU cores on the same (shared-memory) machine to solve the problem faster, but not parallelise across multiple machines.

So I would guess that the best way to parallelise is to run multiple independent jobs on independent nodes of your cluster.

fangohr commented 2 years ago

The second part of your question is a more fundamental issue: simulations with a stochastic noise term bring many challenges. One of them is that you cannot expect exact reproducibility of results (you are using random numbers). Convergence is also not so well defined - what does it mean if you keep adding a random torque onto your equation of motion? The very least you need to do some averaging (over time and/or space) before one can talk about convergence.

This may be a question that is better suited for the mumag mailing list.

MichaelSherburne commented 2 years ago

@fangohr Thank you!

I've been finding that the results for my case are reproducible at a given time step at 300K. Even the magnitudes are the same. OOMMF may be averaging already in this case or could be a special case in the simulation with a single magnetic nanoparticle? However, I've been noticing that the results themselves change with finer time steps, hence why I am curious if going to very small time steps may result in the output changing less drastically. I'm not expecting it to stop changing, but if the difference between one time step to the next is minimal, then it may be getting to convergence. I attached what I have been seeing with smaller time steps below:

1e-13 Time Step image

1e-14 Time Step image

1e-15 Time Step image

As I get down to 1e-15, it is starting to resemble "pulse" response lengths that I would be expecting from another paper using atomistic modeling, but not quite there. If I go down in time steps further, it may start to approximately match. I agree it won't match perfectly, although it would give me more confidence if on average the results are showing something realistic. OOMMF may not be the best code for this, but I am curious in trying to see if it can give a reasonable approximation.