Support for multi-node jobs

[wbinventor]

Currently, OpenMC is simply run using subprocess.call at each timestep. It may be nice to instead use the more customized openmc.run(...) method which has more flexible support for multi-processing/threading.

In addition, some thought should be given to how one might deploy an OpenDeplete/MC job across multiple nodes. The need to be able to use 10s (or even 100s) of nodes is painfully obvious by simply trying to run a deplete a single 17x17 PWR fuel assembly, since OpenMC's runtime is inversely proportional to the number of nuclides in the problem.

@cjosey do you have any recommendations on how to use multiple nodes with your framework? Must something like mpi4py be incorporated to ensure that only a single Python process, perhaps MPI rank 0, is used to manage the simulation workflow?

[cjosey]

Currently, multi-node is limited to just the OpenMC side, but I have been mulling how to handle domain decomposition on the depletion operator.

For this particular problem, here is roughly where communication needs to happen:

1. Each cell reads reaction rates in parallel. Does Pandas dataframe load the entire object from disk at once, or just the slice the user needs? If it's the former, either it will have to change, I'll have to rewrite the HDF5 read of tallies just for this purpose, or only rank 0 reads and communicates.
2. Each node integrates the local power and communicates that to rank 0.
3. Rank 0 communicates normalization factor to all nodes.
4. All cells integrate in time the substep.
5. Final nuclide numbers communicated to rank 0 for XML write or parallel written to HDF5 file for OpenMC to read number densities.
6. (Adaptive timestepping only) During the ATS stage, compute local node relative error. Communicate to rank 0.
7. (Adaptive timestepping only) Rank 0 communicates timesteps / acceptance criterion to nodes.
8. Final result parallel written to HDF5 results file. All I need to do is adjust the chunking and this will work well.

With some modification to OpenMC, this can be made extremely parallelizable. Beyond this, I do not know what the plan is.

[wbinventor]

I was actually pointing to a seemingly much simpler issue. Let me rephrase - how would you submit a job on a cluster? I want to run OpenMC in parallel across nodes, but don't really care about distributed parallelism for OpenDeplete.

But of course the scheme you describe for running the depletion calculation in parallel makes sense to me. Do note for (1) that Pandas does indeed read the entire object from disk, but Dask and (PySpark)[http://spark.apache.org/docs/0.9.0/python-programming-guide.html] are designed for out-of-core applications.

[cjosey]

Ah, in that case, just adjust the openmc call:

settings.openmc_call = ["mpiexec", "openmc"]

Depending on how it's set up, mpiexec should just link to your SLURM/PBS settings and not need be passed arguments. That is how I've been testing this on a supercomputer.

[wbinventor]

@cjosey alright I'll try that. I still think it would be nice to use the openmc.run wrapper to more cleanly parametrize this.

[cjosey]

Closed with #30.