MOOSE Issue: Change to interior Node BCs that Causes Segfaults if NumCores < 2

[aladshaw3]

Testing the latest version of MOOSE on CATS has resulted in some critical errors in how boundaries are now being handled for multi-block domains. This impacts recent CO2 conversion work, as well as some older DG Navier-Stokes work (and breaks a few unit tests).

The issue is derived from certain interior boundary sets not being recognized as part of a particular block. This was seen previously, but the fix was to just reorder how the blocks are constructed in the mesh. However, a change in MOOSE has caused that previous work-around to no longer be valid.

For the time being, when using CATS, DO NOT use INL MOOSE main. Instead, use this fork of MOOSE (https://github.com/aladshaw3/moose/tree/master) when installing MOOSE. This fork should work with CATS.

[aladshaw3]

Issue occurs primarily in test dg_incomp_navier_stokes/serpentine and any of the 3D CO2 conversion files.

With the CO2 conversion files, the issue is specifically for variables C_H and phi_s which have a Integrated type BC at the boundary named cathode_interface_membrane. This boundary is no longer considered a member of the cathode block and instead is being considered part of the catex_membrane block. Niether variable exists on the catex_membrane block.

The solution may be to somehow reassign the boundary to the correct block in the mesh, or add those variables to the other block with a 'dummy' kernel active.

[aladshaw3]

Question: Is it even possible to use Integrated BCs for interior boundaries anymore? (needs to be tested)

[aladshaw3]

Issue occurs primarily in test dg_incomp_navier_stokes/serpentine and any of the 3D CO2 conversion files.

With the CO2 conversion files, the issue is specifically for variables C_H and phi_s which have a Integrated type BC at the boundary named cathode_interface_membrane. This boundary is no longer considered a member of the cathode block and instead is being considered part of the catex_membrane block. Niether variable exists on the catex_membrane block.

The solution may be to somehow reassign the boundary to the correct block in the mesh, or add those variables to the other block with a 'dummy' kernel active.

Current workaround:

• Remove Neumann interface BC for phi_s. It is unnecessary as the 'Natural BC' will suffice.
• Add C_H to the catex_membrane block, along with D_H and Dd_H and allow for C_H to 'flux' through the membrane, while still imposing the flux rate at the interface BC.

[aladshaw3]

New little wrinkle. The reported issues (at least for the CO2 conversion tests) completely disappear if running with more than one core...

Not sure what this means.

[aladshaw3]

## NOTE: Because of some changes in MOOSE, we may need to make modifications
# to our multi-block domains as was done here. Variable C_H was added to the
# catex_membrane domain because we wanted to impose an Integrated BC on that
# variable for a node set that exists between the catex_membrane and the cathode.
# In a prior MOOSE build, these internal boundaries could be assigned to one domain
# or another. In this case, that node set was assigned to the cathode domain. However,
# due to changes in MOOSE it appears that the internal domains are now assigned to
# both domains that share those nodes (if the nodes are shared). This means that
# any variable that we want imposed with an Integrated BC on that node set MUST
# now exist in both domains.

# HOWEVER, after some additional testing, if executing the code with MPI and a
# number of cores > 1, then the issue does not persist. It is uncertain as to
# why this is. It may be a new bug in MOOSE.

# Updates have been made to the primary CO2 electrochem files that were impacted,
# and some older files (except for this one) have been removed. This file will
# serve as notes and a reminder for my future self.

[aladshaw3]

May need to bring this up with MOOSE user's group (bug appears in serial mode, not in parallel)