Proposal: rewrite core codebase in C++

[paulromano]

I often get asked why OpenMC is written in Fortran instead of C or C++ or another language. When I started the code, @bforget and I were interested primarily in developing and studying parallel algorithms for MC transport aimed at overcoming some of the traditional limitations on scalability. As such, we were naturally very interested in Fortran co-arrays, which had recently been added to the language in the 2008 standard and offered the promise of expressing parallelism through RDMA. I started playing around with co-arrays during the summer of 2011 and quickly discovered that there were a number of major shortcomings:

• There was really only one implementation at the time (Intel), and it wasn't just standard in their compiler; you also had to buy their cluster software (can't remember the name). That would be pretty much in diametric opposition to the idea of OpenMC being free and open if you have to buy software just to run in parallel.
• In F2008, co-array functionality lacks collective operations which is a major problem for scalability (supposedly will be fixed in F2015).

That was pretty much the end of our experiments with co-arrays. However, we had a considerable amount of functionality in OpenMC already built on Fortran, and I didn't want to "waste time" as a PhD student just rewriting what I already had. Anyone who has done a thesis knows that expediency often wins.

Fast forward to present day. OpenMC is still in Fortran and has grown from about 14k lines of code (version 0.4.0) to now 40k lines of code. That may sound like a lot, but by most standards 40k lines of code is actually quite small. MCNP 6.1 is something like ~300k, Serpent2 is ~240k, and Geant4 is something like 1M. I believe there are a number of compelling reasons that we should now consider rewriting the core codebase in C++ while the it is still relatively small:

• Use of external libraries: Right now if you want to use a library from OpenMC, you better hope that it has a Fortran API. If not, you better hope that whatever you want to do can be done in a C interoperable manner. Anyone who hasn't been living under a rock knows that most libraries are written in C or C++. Let's take, for example, parallelism, which has been a primary driver in the development of OpenMC. Some of the available options for parallelism in C++ include standard-library features (threads, mutexes, locks, futures, etc.), C++ AMP, Intel TBB, Cilk Plus, Vc library, etc., and that's on top of the usual MPI and OpenMP C interfaces. For Fortran, you really only have MPI and OpenMP. This is not to say that we should be using any of these fancy options, but just merely to point out that we don't have the option even if we wanted to, especially for R&D purposes.
• Rich standard-library: Ever looked at dict_header.F90, list_header.F90, or stl_vector.F90 in OpenMC? All are naive attempts to make up for what the Fortran language is missing -- a standard library with containers. Not to mention that string handling in Fortran is unsatisfying.
• Ability to get closer to hardware: Since OpenMC is primarily a code intended for advanced R&D, a rewrite in C or C++ would allow researchers to study ideas that require leveraging hardware-specific features, which almost certainly means using some C API or otherwise calling a C function. Again, you might be able to do this with Fortran and the C interoperability module, but it may be painful/limited.
• Interoperating with other community codes: There is already interest in collaboration with the INL/MOOSE, ORNL/Shift, and FermiLab/Geant teams. All of these projects are C++-based, so there may be benefits to operating in the same language. PyNE, an open-source NE toolkit, also has a substantial amount of C++ code.
• Moving towards libopenmc: The development team has had a number of discussions recently about moving OpenMC in the direction of a library that could be called by other software. Again, you could do this in Fortran, but it is a lot more error-prone.
• Compiler issues: Now that we are more heavily utilizing F2003 and F2008 issues, we are often running up against compiler bugs and limitations. I've probably reported somewhere around 20 bugs between the gfortran, Intel, PGI, and IBM compilers. A rewrite in C++ won't necessarily mean all problems are gone, especially because I would advocate that we adopt C++11, but some of the more esoteric issues specific to Fortran could be avoided.
• Talent pool available: There are obviously many more C++ developers than Fortran developers, and, depending on who you ask, C++ is usually among the top 5 most popular programming languages. Add to this the fact that Fortran, justly or unjustly, conjures negative feelings for many people.

In terms of the mechanics of a rewrite, some code could immediately be removed (all those containers I mentioned, maybe ~3k lines of code), a sizable chunk could easily be rewritten (recent additions to the code that are already OO-style), and then another sizable chunk will require deeper thought. That being said, a few of the modules that fit in that last category are things we are already targeting a refactor for anyway, e.g., geometry and tallies. I don't want to understate the level of effort and time required, but I do think that it might not be as bad as one might naturally think, especially if we're able to get a few people involved.

Some of the risks involved:

• C++ is a large and complex language. There is no doubt about that. When I think about students looking to use OpenMC for their research, they will certainly have to go through a steep learning curve if they haven't used C++ before. But, I think that their time will be better spent learning C++ than learning Fortran and they will end up with a language skill that is beneficial outside of just scientific computing.
• Another language pops up that is way better than C++ and satisfies most of our desires. For example, there are good arguments that Julia, young as it is, is a better language for scientific computing. However, rewriting in Julia would be a lot riskier than C++ in my opinion. The same is likely to be true of other new languages as well.

[smharper]

:+1: :+1: Yes, I am on-board 100% with this change. The only major question I have in my mind is, should we try to rewrite the whole thing in one big, painful surge, or should we rewrite it chunk-by-chunk using C-interoperability to keep running with a mixed code base?

[smharper]

Also, I want to mention one other benefit with moving to C++. Python has a well-used C/C++ API so moving to C++ will allow us to call some of our core routines directly from the Python API.

[paulromano]

@smharper In my opinion, the second option (chunk-by-chunk using C-interoperability) actually sounds more painful and restrictive.

[mellis13]

Having experienced the limitations of C interoperability while working on the coupling of MOOSE to OpenMC, rewriting OpenMC in C++ would be a great help in multiphysics coupling. This is probably too late to drastically change the course of my PhD work, but I think it opens up a lot of future research opportunities with other major codes. Great suggestion, @paulromano !

[nelsonag]

I am absolutely a fan of this idea. This also enables easier GPU algorithm development without having to pay for the PGI compiler. The only down side is I have now been working with Fortran so long that I am likely very rusty with C++!

I agree with the thought by @paulromano that chunk-by-chunk can be pretty painful and so doing it at once is preferred. However, how would we go about doing it all at once? Basically do a fork, hold all pull requests not C++ related on this repo until the fork is completely moved over to C++? Then, if all tests and example problems pass, we move on? If this is the model, when should we enable the feature pause?

Edit: removed the gif (my eyes hurt every time I come here now... sorry), and got rid of backticks around feature pause.

[wbinventor]

I don't think this comes as a surprise, but I also would be a big supporter of a move to C/C++. I agree with each of @paulromano's reasons to do so. I would extend the "talent pool" reason to include the argument made for the first risk - to paraphrase, any student is better off spending the late night hours of their PhD learning the nuances of C++ rather than Fortran from a career standpoint. This is also true for any aspiring PhD graduates out there who may be debating how and where to spend the next chapter of their career :-)

[smharper]

@nelsonag, we don't need a whole other fork, we can just do all of our C++ conversion in a different branch. We must name this branch "OpenMC++". When we're done, we can then merge it over to develop

[walshjon]

Adding to the apparent consensus, I'll just underscore @paulromano 's and @wbinventor 's points regarding the value of the proposed switch to future grad students. I can foresee this delaying the integration of some features currently under development, but that will always be the case. It seems to me that now is as good of a time as any to rip the band-aid off.

[paulromano]

Great to see that so many are on board with this proposed change. As @walshjon points out, this will certainly cause some pain with respect to integrating features that are currently under development, but I think as long as we are careful with planning, hopefully we can avoid any major problems. For now, don't worry about feature freeze or anything like that -- continue doing whatever you were doing before. Once I have more clarity about our internal plans at ANL, we can continue the discussion about planning.

[siegela67]

I would like to disagree but I can’t. To me the biggest concerns are related vendor support, especially on the cutting edge of HPC. Fortran is starting to fall behind quickly. At a recent AMD meeting one of their principle architects asked the audience (computational scientists) if anyone was still using fortran. Quite a few raised their hand but it was clear that Fortran support wasn’t high on AMD’s priority list.

Fortran is also a far weaker language in general. Twenty years ago many more codes were structured grid, where conformable and easy to optimize (for compiler) 3D arrays were the main data structure, codes were small enough where abstraction wasn’t as important. Fortran had some advantages in those days. Scientists also had much less expertise in computer science, so Fortran was seen as a way to insulate domain specialists from both hardware on one end and (unnecessarily) bloated software architecture on the other. But modern hardware made loop based indexing in C equivalent to fortran performance wise, unstructured meshes are far more common (more metadata), and codes got much more complex and came to rely more on abstraction, which Fortran is horrific at (user defined types are awful). I think C++ would be a much better language especially for OpenMC.

That said I’ve studied dozens of HPC C++ apps in my life. Some are among the worst architected codes I’ve ever seen. In the hands of an inexperienced programmer C++ just multiplies the ways to hang yourself. So I would start with at least a basic set of coding standards and be a little bit strict about code review.

On Mar 2, 2016, at 1:41 PM, Paul Romano notifications@github.com wrote:

I often get asked why OpenMC is written in Fortran instead of C or C++ or another language. When I started the code, @bforget and I were interested primarily in developing and studying parallel algorithms for MC transport aimed at overcoming some of the traditional limitations on scalability. As such, we were naturally very interested in Fortran co-arrays, which had recently been added to the language in the 2008 standard and offered the promise of expressing parallelism through RDMA. I started playing around with co-arrays during the summer of 2011 and quickly discovered that there were a number of major shortcomings:

There was really only one implementation at the time (Intel), and it wasn't just standard in their compiler; you also had to buy their cluster software (can't remember the name). That would be pretty much in diametric opposition to the idea of OpenMC being free and open if you have to buy software just to run in parallel. In F2008, co-array functionality lacks collective operations which is a major problem for scalability (supposedly will be fixed in F2015). That was pretty much the end of our experiments with co-arrays. However, we had a considerable amount of functionality in OpenMC already built on Fortran, and I didn't want to "waste time" as a PhD student just rewriting what I already had. Anyone who has done a thesis knows that expediency often wins.

Fast forward to present day. OpenMC is still in Fortran and has grown from about 14k lines of code (version 0.4.0) to now 40k lines of code. That may sound like a lot, but by most standards 40k lines of code is actually quite small. MCNP 6.1 is something like ~300k, Serpent2 is ~240k, and Geant4 is something like 1M. I believe there are a number of compelling reasons that we should now consider rewriting the core codebase in C++ while the it is still relatively small:

Use of external libraries: Right now if you want to use a library from OpenMC, you better hope that it has a Fortran API. If not, you better hope that whatever you want to do can be done in a C interoperable manner. Anyone who hasn't been living under a rock knows that most libraries are written in C or C++. Let's take, for example, parallelism, which has been a primary driver in the development of OpenMC. Some of the available options for parallelism in C++ include standard-library features (threads, mutexes, locks, futures, etc.), C++ AMP, Intel TBB, Cilk Plus, Vc library, etc., and that's on top of the usual MPI and OpenMP C interfaces. For Fortran, you really only have MPI and OpenMP. This is not to say that we should be using any of these fancy options, but just merely to point out that we don't have the option even if we wanted to, especially for R&D purposes.

Rich standard-library: Ever looked at dict_header.F90, list_header.F90, or stl_vector.F90 in OpenMC? All are naive attempts to make up for what the Fortran language is missing -- a standard library with containers. Not to mention that string handling in Fortran is unsatisfying.

Ability to get closer to hardware: Since OpenMC is primarily a code intended for advanced R&D, a rewrite in C or C++ would allow researchers to study ideas that require leveraging hardware-specific features, which almost certainly means using some C API or otherwise calling a C function. Again, you might be able to do this with Fortran and the C interoperability module, but it may be painful/limited.

Interoperating with other community codes: There is already interest in collaboration with the INL/MOOSE, ORNL/Shift, and FermiLab/Geant teams. All of these projects are C++-based, so there may be benefits to operating in the same language. PyNE, an open-source NE toolkit, also has a substantial amount of C++ code.

Moving towards libopenmc: The development team has had a number of discussions recently about moving OpenMC in the direction of a library that could be called by other software. Again, you could do this in Fortran, but it is a lot more error-prone.

Compiler issues: Now that we are more heavily utilizing F2003 and F2008 issues, we are often running up against compiler bugs and limitations. I've probably reported somewhere around 20 bugs between the gfortran, Intel, PGI, and IBM compilers. A rewrite in C++ won't necessarily mean all problems are gone, especially because I would advocate that we adopt C++11, but some of the more esoteric issues specific to Fortran could be avoided.

Talent pool available: There are obviously many more C++ developers than Fortran developers, and, depending on who you ask, C++ is usually among the top 5 most popular programming languages. Add to this the fact that Fortran, justly or unjustly, conjures negative feelings for many people.

In terms of the mechanics of a rewrite, some code could immediately be removed (all those containers I mentioned, maybe ~3k lines of code), a sizable chunk could easily be rewritten (recent additions to the code that are already OO-style), and then another sizable chunk will require deeper thought. That being said, a few of the modules that fit in that last category are things we are already targeting a refactor for anyway, e.g., geometry and tallies. I don't want to understate the level of effort and time required, but I do think that it might not be as bad as one might naturally think, especially if we're able to get a few people involved.

Some of the risks involved:

C++ is a large and complex language. There is no doubt about that. When I think about students looking to use OpenMC for their research, they will certainly have to go through a steep learning curve if they haven't used C++ before. But, I think that their time will be better spent learning C++ than learning Fortran and they will end up with a language skill that is beneficial outside of just scientific computing.

Another language pops up that is way better than C++ and satisfies most of our desires. For example, there are good arguments that Julia, young as it is, is a better language for scientific computing. However, rewriting in Julia would be a lot riskier than C++ in my opinion. The same is likely to be true of other new languages as well.

— Reply to this email directly or view it on GitHub.

[wbinventor]

I'm currently at a talk on the Julia language with @samuelshaner and @geogunow, and hearing the entire room laugh at us as a few of the remaining Fortran programmers in the world epitomizes this issue.

[siegela67]

how many codes using Julia use 1M cores to do productive science?

On Mar 31, 2016, at 11:11 AM, Will Boyd notifications@github.com wrote:

I'm currently at a talk on the Julia language with @samuelshaner and @geogunow, and hearing the entire room laugh at us as a few of the remaining Fortran programmers in the world epitomizes this issue.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub

[siegela67]

“a few of the remaining fortran programmers” is a wild exaggeration. Look at the metrics of fortran use at the major supercomputer centers. who do they think maintains/develops those codes? Not saying I disagree with language choice change but no need to exaggerate circumstances.

On Mar 31, 2016, at 11:11 AM, Will Boyd notifications@github.com wrote:

I'm currently at a talk on the Julia language with @samuelshaner and @geogunow, and hearing the entire room laugh at us as a few of the remaining Fortran programmers in the world epitomizes this issue.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub

[wbinventor]

I'm not (yet) a huge Julia fan myself, but seeing the languages that folks use at these talks illustrates just why moving to C++ may be important to foster a sustainable base of developers for the future. And of course the programmers at a Julia talk are not representative of HPC in general.

[stangmechanic]

I'd also support this. I notice it's an old thread, any movement? I'd like to pitch in.

Also, I've never been a fan of C++ (I think it's too smart for me). I'd support the idea of keeping it more C like, only using C++ when necessary.

[smharper]

Looks like this move is picking up steam with #996 and #1004. Just want to stick an FYI here to avoid duplication of work: I have a branch here where I'm working on moving the vast majority of the geometry-related code to C++

[paulromano]

Thanks for the heads up @smharper. I also have a branch where I've made the Particle derived type interoperable with a C struct so that it is possible to pass a particle back and forth between C and Fortran. I'll put in a PR for that one soon.

[paulromano]

I created a project board to keep track of who is doing what.