permcody
commented
7 years ago I've all but given up on using the nodal behavior of the FeatureFloodCount object. I'm pretty sure it (flood_type = NODAL) currently doesn't work in parallel at all (even on ReplicatedMesh), and I don't have anyone beating down my door to make it work. The use case for this "topologically connected thingy finder" is all element based and seems much more robust that way.
That doesn't mean that we shouldn't fix the libMesh function and document it, but don't let the current usage in that object drive any requirements for you. I'm half tempted to just remove that code path.
On another note, I'd love to actually test this whole FeatureFloodCount and GrainTracker on a massive problem in the (nearish-)future and get another publication out on it. I'd like to put you on there as an author since a lot of the mesh capability is yours. I tested the GrainTracker on about 6000 grains in 3D and it worked perfectly on ReplicatedMesh back in May but that took an enormous amount of memory. At the time, the DistributedMesh support just wasn't there yet so I didn't even try it. I think we're pretty close to making that all work.
roystgnr
jwpeterson
Copying @permcody since he's futzed with the MOOSE phase_field/src/postprocessors/FeatureFloodCount.C enough to be familiar with the use case that broke for me.
MeshTools::find_nodal_neighbors() has two problems:
1) It produces questionable results when operating on non-first-order geometric elements. The test for neighbors stops at the first encountered distinct node which shares an edge with the input node, and will thus miss any other nodes on the same edge. This would be a reasonable implementation of "find_vertex_neighbors()", because libMesh always orders vertex nodes before mid-edge nodes... but based on the special case behavior @jwpeterson added for EDGE2, EDGE3, and EDGE4, what we really want here is the closest nodes along an edge, not the vertex nodes.
2) With the current libMesh default ghosting, in AMR cases it doesn't do what FeatureFloodCount wants: giving a list of neighbors who are also semilocal. Imagine four equal-h-level QUAD4 elements meeting at a point, the "southwest" of which is then refined. Suppose the southeast quad has the same processor ID as the node. Then, on that processor we are guaranteed to have ghosted: the northeast element and the two adjacent out of the 4 southwest active elements. However, the southwest node of the northwest element (which is not currently guaranteed to be ghosted, since it isn't a side neighbor of the southeast, only a point neighbor!) does not belong to any of those. However, if the northwest element is geometrically ghosted (either because of a geometric ghosting functor which isn't also set as an algebraic ghosting functor, or just because we're on ReplicatedMesh), then find_nodal_neighbors will return that non-semilocal node.
I can probably work around this problem in MOOSE by simply testing for algebraic semilocality in the flood fill algorithm, but we should probably decide what the correct behavior in libMesh is, document it, and make find_nodal_neighbors() behave.