call `WeightedSync` on emf before computing rhs

[BenWibking]

Describe the proposal It appears necessary to ensure that the emf values are identical on edges shared between multiple boxes before computing the magnetic induction equation right-hand-side (rhs).

This can be done with either the OverrideSync or WeightedSync MultiFab functions: https://amrex-codes.github.io/amrex/docs_html/Basics.html#owner-mask.

Describe alternatives you've considered If this is not done, there will be roundoff level differences in the emf on edges that should have identical emfs, which will make the simulation dependent on the grid layout used. This can cause spurious (but small) magnetic field divergence errors.

Additional context This is done in Athena++ (although it may be of greater importance there due to their use of curvilinear coordinates, see section 2.1.4: https://ui.adsabs.harvard.edu/abs/2020ApJS..249....4S/abstract):

• "Even without mesh refinement, numerical errors can cause a slight mismatch between the EMFs on shared edges between MeshBlocks. With the constrained transport scheme, such errors never disappear once generated. This problem becomes more prominent when more complex grids with nonuniform mesh spacing and/or curvilinear coordinates are in use. Moreover, aggressive compiler (non-ANSI-conformant) optimizations can introduce and exacerbate differences associated with roundoff errors. Therefore, the EMF correction step is applied even when mesh refinement is not used. In this case, the EMFs on two shared edges are replaced with the arithmetic average of their values."

[BenWibking]

@AstroKriel I wanted to check if you were already doing this in your code. If not, I think it should be done. This does require creating temporary MultiFabs for the emfs (but not for anything else).