search

OpenCMISS / iron

Source code repository for OpenCMISS-Iron
9 stars 62 forks source link

Domain decomposition issue with nodes and mesh components #102

Open chrispbradley opened 6 years ago

chrispbradley commented 6 years ago

There is an issue with assigning domain boundary nodes to domains and exfiles. Currently nodes that are domain boundary nodes in a domain can be moved to domain ghost nodes in order to improve the load balancing between ranks. The issue, however, is that this is done on each mesh component separately. It is thus possible for a node to be domain boundary node in one mesh component and a domain ghost node in another mesh component. Now, this is not normally a problem as it is the dofs that are important. However, for exfiles which are nodal centric this is a problem because the node that is in the rank for a component and off the rank for another will cause a new header to be generated and for the node to appear in two different .part#.exnode files. Reading them in these different .part# files could cause the nodal values to be overwritten.

rchristie commented 6 years ago

Are you intending to write only nodes x fields on each rank to the .part# file, i.e. internal and boundary, but not ghost? If each field's DOFs are fully on a given rank x node, the EX format can handle this if the header changes whenever the mix of fields on that node x rank changes. Changing headers does make the EX file quite large and inefficient to write, admittedly. It would be a problem if different components for a given field used different mesh components with differing node assignments (internal/boundary/ghost)... is that the issue here? Separately, I had thought an MPI request transferred the ghost DOFs so they should be in sync anyway? If so it doesn't matter which .part# file is read. Also, are you talking about reading them into Zinc/Cmgui or back into Iron?

chrispbradley commented 6 years ago

Currently only the internal and boundary values are written out in each .part# but I guess the ghosts could be as well. The problem is that only certain field components have the different node (which does have different assignment between assignments). These are to be read in to cmgui (i.e., exported). Because some nodes do not have all they components (quads and linears) dummy values i.e., 1.2345678 are written out and so it is not really related to keeping the ghosts in sync via MPI. Writing out the ghosts values as well might be the answer.