Implement multi-group capability in OpenMC

[nelsonag]

As a code geared strictly towards education, it would be useful if OpenMC had a multi-group capability. This MG kernel would allow for a simpler verification of new methods while also enabling an easy transition of new methods from the MG to CE space.

Before this could be attacked, it would be prudent to discuss the input multi-group library format. The ACE multi-group transport tables are nice and all, but I suspect it is both too complicated for our needs and too limited (if one is interested in studying the current approximations made in multi-group theory). I personally would prefer a custom format that we would be in complete control of, with the help of python scripts to convert from the other popular formats to our own.

[nelsonag]

Just an FYI: I have been dabbling with this on my spare time. I came to the conclusion that implementing it correctly in to OpenMC would best be done with polymorphic objects and a more object-oriented design. Of course, that is a little bit more cumbersome with gfortran 4.6.2 then any of us would like. Given that I still had questions I wanted an answer to before the above could come to fruition, I created a branch (https://github.com/nelsonag/openmc/tree/omg) written which strips out the continuous energy portion and leaves behind a lean MG-machine. Its definitely not production level yet (as I only have a few hours in to it) but it does give correct answers for the one problem I've tried so far (a C5G7 pin cell..).

[wbinventor]

Thanks for updating us on the status I also think it of this @nelsonag - I also think it would be nice to have this capability, and am glad to know that you have a starter implementation on your fork. When you are further along you might add an example input file showcasing how to use MGXS - perhaps for the C5G7 pin cell?

As for the object-oriented design, my guess is that @smharper's tally refactorization would also benefit from the use of a newer version of gfortran which supports more OO features. Furthermore, from my experience with PR #379, I found that 4.8 supports more of what I consider to be "boiler-plate" OpenMP than the 4.6 which is the standard for Ubuntu 12 and hence Travis. Perhaps you might develop the "omg" branch with the assumption that >=4.8 will soon(ish) become the standard for Travis and a pre-requisite for OpenMC? Although from what I've read online, it looks like Travis is migrating a Docker-based containerized environment which should make the transition to a more modern Fortran standard if and when we choose to do so.

[nelsonag]

@wbinventor, I will do so. I've got the 2D c5g7 problem running and I am reproducing the reference eigenvalue w/in one standard deviation, so that's great news. Working on the power and flux distributions now and first have to fix some issues I uncovered in either tallying or in the interaction of the post-processing with those tallies. But once I get that done and verified I will post the inputs.

On that note, do you (@wbinventor) know where to find the C5G7 reference results? I can't even find a form to fill out at the OECD to request the full cd.

Thanks, Adam

On Wed, May 20, 2015 at 2:33 PM, Will Boyd notifications@github.com wrote:

Thanks for updating us on the status I also think it of this @nelsonag https://github.com/nelsonag - I also think it would be nice to have this capability, and am glad to know that you have a starter implementation on your fork. When you are further along you might add an example input file showcasing how to use MGXS - perhaps for the C5G7 pin cell?

As for the object-oriented design, my guess is that @smharper https://github.com/smharper's tally refactorization would also benefit from the use of a newer version of gfortran which supports more OO features. Furthermore, from my experience with PR #379 https://github.com/mit-crpg/openmc/pull/379, I found that 4.8 supports more of what I consider to be "boiler-plate" OpenMP than the 4.6 which is the standard for Ubuntu 12 and hence Travis. Perhaps you might develop the "omg" branch with the assumption that >=4.8 will soon(ish) become the standard for Travis and a pre-requisite for OpenMC? Although from what I've read online, it looks like Travis is migrating a Docker-based containerized environment http://blog.travis-ci.com/2014-12-17-faster-builds-with-container-based-infrastructure/ which should make the transition to a more modern Fortran st andard if and when we choose to do so.

— Reply to this email directly or view it on GitHub https://github.com/mit-crpg/openmc/issues/336#issuecomment-103988096.

[wbinventor]

Are you referring to the reference pin powers? If so, I have them in both XLSX and HDF5 formats. I'll send them along to you offline. The reference eigenvalue is in the literature (1.18655 +/- 10 pcm). On May 27, 2015 6:21 AM, "Adam Nelson" notifications@github.com wrote:

@wbinventor, I will do so. I've got the 2D c5g7 problem running and I am reproducing the reference eigenvalue w/in one standard deviation, so that's great news. Working on the power and flux distributions now and first have to fix some issues I uncovered in either tallying or in the interaction of the post-processing with those tallies. But once I get that done and verified I will post the inputs.

On that note, do you (@wbinventor) know where to find the C5G7 reference results? I can't even find a form to fill out at the OECD to request the full cd.

Thanks, Adam

On Wed, May 20, 2015 at 2:33 PM, Will Boyd notifications@github.com wrote:

Thanks for updating us on the status I also think it of this @nelsonag https://github.com/nelsonag - I also think it would be nice to have this capability, and am glad to know that you have a starter implementation on your fork. When you are further along you might add an example input file showcasing how to use MGXS - perhaps for the C5G7 pin cell?

As for the object-oriented design, my guess is that @smharper https://github.com/smharper's tally refactorization would also benefit from the use of a newer version of gfortran which supports more OO features. Furthermore, from my experience with PR #379 https://github.com/mit-crpg/openmc/pull/379, I found that 4.8 supports more of what I consider to be "boiler-plate" OpenMP than the 4.6 which is the standard for Ubuntu 12 and hence Travis. Perhaps you might develop the "omg" branch with the assumption that >=4.8 will soon(ish) become the standard for Travis and a pre-requisite for OpenMC? Although from what I've read online, it looks like Travis is migrating a Docker-based containerized environment < http://blog.travis-ci.com/2014-12-17-faster-builds-with-container-based-infrastructure/

which should make the transition to a more modern Fortran st andard if and when we choose to do so.

— Reply to this email directly or view it on GitHub https://github.com/mit-crpg/openmc/issues/336#issuecomment-103988096.

— Reply to this email directly or view it on GitHub https://github.com/mit-crpg/openmc/issues/336#issuecomment-105905471.

[nelsonag]

I have this thing up and running and looking pretty good. Just ran the 3D C5G7 (unrodded) problem and am matching eigenvalues quite well (if I remember right - not at home now - I matched the published KENO MG results and we were both w/in a st dev of the reference MCNP results).

I have some time Friday to dig in to actually merging this in to upstream. I'd rather not waste my time though, so before I do this, is there substantial enough interest in the addition of the Multi-Group capability to OpenMC? I'd really settle this up front because this could be a non-negligible amount of work long term in sustaining this capability.

If the answer to the above is 'yes, we are interested', the next question is the 'how would you like it represented in the code itself?' The following are my thoughts on that right now.

Most of the nuts and bolts changes to the code will be at the transport routine and in some of the routines it calls (update_xs, most of physics.F90). Since this is the case I can either implement this in one of two ways:

1. Just keep the MG capability in separate files and make use of if/elses at a very high level (probably at the transport.F90 level and the I/O & pre-processing stuff)
   • Pros:
     • the code for CE and MG would be much cleaner and easier to follow than option 2;
     • shouldn't rely much on compilers supporting standards they claim to support (looking at you gfortran 4.6.2);
     • should have best performance as you only need to undergo the if/then query as much as once per batch or as few as once per execution.
   • Cons:
     • inevitable duplication of code
     • increased risk of having one side of the code (CE or MG) being updated with new functionality and the other side being neglected.
2. Keep changes only where they need to be. This can be done by implementation of objects + polymorphism, function pointers, and good ol' if-then blocks.
   • Pros:
     • Less duplication of code;
     • harder to forget about the MG code since its staring you right in the face instead of hiding in a corner;
   • Cons:
     • Possibility the implementation could get confusing, especially when targeting 4.6.2;
     • not sure the degree that branching overhead could be minimized, so we'll assume there is the potential for a few queries on if MG or not per collision or history.

[nelsonag]

On thinking about this more, I realized I do have a presentation to write for ANS DC and so I might not get to this Friday. I'd still like to have this discussion though.

[paulromano]

FWIW, my approach for photon transport was more akin to option 1. However, overall I think we should be moving more in the direction of option 2 for exactly the reasons you've mentioned, namely the fact that it reduces code duplication and decreases the chances that code gets out of sync (exactly what happened with eigenvalue and fixed source modules -- that's why I merged them into simulation.F90).

It's hard to know what exactly the impact of OO overhead from virtual dispatch, etc. is until you actually do it. In my recent refactor of surfaces, which is now very OO, I was able to pretty much get almost the same performance, which came as a little bit of a surprise. It can also be very compiler- and even system-dependent.

For your case, what do you lose by target gfortran 4.6 as opposed to 4.8? I remember running into something but can't remember what exactly it was.

[paulromano]

To answer your more general question on the value of MG in OpenMC, I think it is a worthwhile addition for academic uses as well as deterministic comparisons (especially in light of the capabilities introduced in #471). I do worry about maintainability, but I don't think that should stop us from doing it.

Is your capability based on MG ACE files?

[smharper]

+1; I would also like to see this capability in OpenMC.

[wbinventor]

I also think this would be a nice capability to include. It doesn't seem to me that multi-group MC would ever be competitive with an efficient deterministic counterpart (perhapshw this is debatable). But if nothing else (and it's a lot) it would be a nice pathway to benchmark said deterministic codes.

[nelsonag]

@paulromano: I need to think about what the specific gfortran 4.6-4.8 issues were. This harkens back to my MPACT days and I think there was a PR/issue on here too that was complaining about it as well. The capability is not based on MG ACE files; since I am mostly interested in creating my own mgxs, I created my own format (https://xkcd.com/927/). I think it matches the style of the rest of OMC well though, as its an XML based file. For me this significantly reduced my overhead (I could put in the format what I needed - and XML is easy to read and write to - and not have to worry about populating superfluous data to the task at hand just because some standard requires it).

@paulromano & @wbinventor: Yeah I think that is one major benefit; the ability to compare and contrast with deterministic methods in a more apples-apples world. Others include: ability to research hybrid methods in a closer to in-line system (at least right after this potential PR, the same geometry.xml file can be used for the CE MC and MG MC modes), and finally, though this is out of scope right now, I have a gut feeling (but not much more than that), that MG MC could be more scalable on advanced architectures than deterministic methods when were talking about larger 3D problems.

[wbinventor]

I'll put in my two cents that a format based on HDF5 might better suited than XML. It seems that a lot of new crazy high-fidelity features coming out in the code are using HDF5 not just for output but also for input (@cjosey might have an opinion on this). I think there many, many reasons to use HDF5, not least of which is that an ASCII-based format is ill-suited to store lots of arrays of floating point data in a memory-efficient way.

[nelsonag]

I agree completely. Right now I have the xml interface because I started before our hdf5 interface was to the state it is now and I had less experience with it. I think I would prefer to do it with HDF5 now and may come out of this with an hdf5 implementation. That being said, part of me likes the idea of an ascii-based system too simply for ease of tweaking on toy problems.

On Wed, Oct 28, 2015 at 5:14 PM, Will Boyd notifications@github.com wrote:

I'll put in my two cents that a format based on HDF5 might better suited than XML. It seems that a lot of new crazy high-fidelity features coming out in the code are using HDF5 not just for output but also for input ( @cjosey https://github.com/cjosey might have an opinion on this). I think there many, many reasons to use HDF5, not least of which is that an ASCII-based format is ill-suited to store lots of array-based floating point data in a memory-efficient way.

— Reply to this email directly or view it on GitHub https://github.com/mit-crpg/openmc/issues/336#issuecomment-151991980.

[paulromano]

+1 for no ACE. Carries too much baggage at this point. Also I would encourage you to do HDF5 as I'm hoping all our nuclear data files will move in that direction. It's true ASCII/XML is good for toy problems, but HDF5 is simple enough to make mods too as well. Certainly way easier than ACE...

[nelsonag]

I did get this started. So far I have implemented the MG capability for everything short of tallying, output, and CMFD (im sure there will be surprises too). Right now I get the same keff with my separate MG code as I do in this combined version. Thats great news. One thing that is not good news is that I have an OpenMP dependent bug. Those are the worst.

If anyone is curious how I'm implementing it, feel free to check out my mg branch. I am going for a strategy of get something implemented that is close to right, but not fully optimized to not disrupt the CE runtimes. They look pretty close at the moment, but not perfectly.

[nelsonag]

Closed via #494