Problems with scalar-expressions and logical-expressions in SMP mode

[mabruzzo]

Overview

There are some issues with using a parameter file including scalar-expressions and logical-expressions in the parameter file when Enzo-E is a compile with a build of Charm++ that uses SMP mode. I think I replicated this problem about a year ago, but I heard about this issue from @jobordner, first.

As I recall, this the issue is related to thread-safety/reentrancy problems with either the parser or the ExprEval library (which actually evaluates the expressions).

Consequences

As far as I can tell, this affects 5 separate pieces of functionality:

1. The "value" initial conditions (used to help initialize test problems using algebraic expressions)
   • In the documentation there is a warning that this initializer "does not work reliably for multi-node problems." It's not obvious clear to me whether this is documenting the same issue being raised here or if there are separate that affect this even when using charm++ without SMP-mode.
2. The "vlct_bfield" initial conditions (assists with initializing face-centered magnetic fields for use with the VL+CT integrator)
   • Specifically, issues can arise when scalar-expressions are used to specify components of the magnetic vector potential (which are then used to automatically initialize divergence-free magnetic fields)
   • This initializer also supports some separate additional functionallity that is unaffected by SMP mode.
3. The "mask" type of refinement criterion. This is specified in the Adapt section of the parameter file and implemented with the RefineMask class.
4. The "inflow" type of boundary condition. This is specified in the Boundary section of the parameter file and implemented with the BoundaryValue class.
5. The general masks that can be specified in the Boundary section of the parameter file that facilitate the specification of multiple boundary objects that act on separate areas of a particular boundary.

Because evaluation of scalar-expressions and logical-expressions is relatively slow, the recommended best practice is to use specialized problem initializers for large production science simulations (instead of the "value" and "vlct_bfield" initializers). So in a sense, the issue with these initializers doesn't seem too problematic. At the same time, these initializers are used in a large fraction (if not the majority) of test problems. Thus, this issue significantly reduces test coverage in SMP mode (and potentially hides other existing issues).

The issues for the other functionallity might cause problems in SMP mode for:

• simulations using Static Mesh Refinement
• wind tunnel simulations

Outstanding Questions

It might be useful to assess the extent of the problem.

• Does this affect masks that are specified with a .png file? (I'd assume the answer is "no")
• Are there any issues when a parameter that could accept a scalar-expression is just passed a single floating point value? (I'm pretty fuzzy on this, but I think I did some basic tests and found that the answer was "no").

If the problem is not easily addressable, maybe we modify Enzo-E/Cello to exit with an error message if the user tries to use this functionallity and the code was compiled in SMP mode.

[mabruzzo]

Just wanted to add a brief update: I don't think the ExprEval library is actually used by Enzo-E