search

OpenCMISS / iron

Source code repository for OpenCMISS-Iron
9 stars 62 forks source link

Ghost Dofs incorrectly defined as internal on home rank. #169

Open farg967 opened 5 years ago

farg967 commented 5 years ago

This problem is seen when running monodomain_2D_HH in parallel.

I have narrowed the main cause of the problem down to DOMAIN_MAPPINGS_NODES_DOFS_CALCULATE, line 4554

IF(NUMBER_OF_DOMAINS==1) THEN

should be

IF(MAX_NUMBER_DOMAINS==1) THEN

This bug causes boundary nodes/dofs one element away from the boundary plane between computational domains to be incorrectly defined as internal dofs. I think the problem originates from a double up on the meaning of "boundary". In some places, "boundary" means it is another domains ghost. In other places, "boundary" means it is on the physical boundary plane between ranks. Unless advised otherwise I will use "boundaryPlane" for the latter from now on.

There is a bit more to the fix than the above line, which I am working on atm.

Note: This will only be a temporary fix, as NODES_DOFS_CALCULATE will be done locally soon.

farg967 commented 5 years ago

Fixed, with above commit.

chrispbradley commented 5 years ago

Hi Finbar, just reopening the issue as it gets closed when the pull request containing the commit is pulled into the main trunk rather than when you do the commit.

lorenzo-mechbau commented 5 years ago

I further modified the routine DOMAIN_MAPPINGS_NODES_DOFS_CALCULATE and pushed it in the branch I am currently working on, FV_merged. Now the outcome is correct also for a number of ranks >2, keeping Finbar's improvements.

Naming convention: All nodes can be either internal or involved in communication, that is nodes on boundary elements. The latter can be in turn either boundary nodes, defined as LOCAL_DOMAIN_BOUNDARY, i.e. ghost-send, or ghost nodes (ghost-receive). Additionally, some of the nodes on boundary elements are boundary plane nodes, that is they can be found on the physical boundary plane between ranks, as Finbar wrote.

See also the project: https://github.com/users/lorenzo-mechbau/projects/1