Faces can have their element indices swapped when moving when load balancing.

[guillaumetousignant]

This is a race condition. The following happens: Face 86 has elements [78, 56], with sides [3, 1]. Element 78 is moved to 100, and element 56 is moved to 78. The 78 coming up twice causes the race condition.

If element 78 is executed first, it becomes [100, 56], sides [3, 1], then [100, 78], sides [3, 1]. This is the expected result.

If element 56 is executed first, it becomes [78, 78], sides [3, 1]. Then element 78 can find itself in either positions. In this case the check is made such that it checks the right spot first, therefore becoming [78, 100], side [3, 1]. This is backwards, the sides are wrong, as will be the normal and everything.

[guillaumetousignant]

A bad solution is to also check if the side matches, which will fix this case. It won't work if both elements present the same side index to the face. As the meshes generated by the mesh generator all have the same rotation, this can't happen with these meshes. Very bad solution.

[guillaumetousignant]

The faces should update their own element indices, and the elements should update their own face indices. This will take a while.

[guillaumetousignant]

Took forever but now elements calculate the new faces indices themselves, same for faces. recv_mpi_boundaries_destinations_reuse_faces is the only exception, but it seems to work still.