Ancestor tracking not working with multiple cores

joeweaver commented 2 years ago

The lineage-tracking branch has been pushed. It contains prototype code which uses a fix_property to track the ancestor id of all cells. Every initial cell in the system is its own ancestor. Upon division, the newly created cell ought to have the same ancestor as its sister/mother.

This works in one core. When visualized, each blob of cells has an ancestor corresponding to the tag of the initial cell. (n.b. color is set to ancestor, visualization has EPS particles filtered out, all cells shown, sphere glyph is scaled by 1*diameter.)

no_mpi

When used with multi-cores (confirmed for mpirun -np 2 and mpirun -np 6), the blobs start off fine, but eventually we start seeing cells down the centerline (dark blue spheres down the middle) which do not have the correct ancestor. Further the ancestor appears set to 0.

mpi6

In debugging attempts, I've altered the calculation of the ancestor id to determine if it was getting copied over wrong or an initialization issue:

When initial bugs are given an ancestor id of tag +10 instead of tag under FixPropertyAncestor::init(), the mis-labelled bugs remain at 0.
When the newly created cell is given an ancestor property of -1 instead of 0 under FixPropertyAncestor::set_arrays(), there is also no change.
When the newly created cell is given an ancestor property of of _sister's ancestor_ + 5 in FixPropertyAncestor::update_arrays(), the first set of incorrect bugs remain at ancestor 0.

Given that none of those influence the change, that this only occurs with MPI, and it is spatially based on a centerline, I believe the bug has something to do with how cell properties are managed across spatial partitions. Not sure if it's a ghost cell thing, partition communication thing, or cell migration thing.

Once we know the root cause and have a fix, we should also check FixPropertyCycletime and FixPropertyGeneration to see if they're affected.

shelllbw commented 2 years ago

Could you upload the example to the branch so that I can use to reproduce the problem?

joeweaver commented 2 years ago

Sure. I've pushed a new example to the lineage-tracking branch. Should be under examples/property-nufeb/Inputscript-lineage.lmp

shelllbw commented 1 year ago

The issue has been fixed in commit 9959103e2afbef3891ff01ee1b79bb0ed4ed315c

nufeb / NUFEB-2

Ancestor tracking not working with multiple cores #7