Closed tamiko closed 2 months ago
Mixed refinement/coarsening should, in my opinion, not result in any significant difference compared to doing local refinement only. It obvious does lead to different numberings of vertices, the cells are traversed differently (because new cells might get inserted both at the end or in the middle when free space comes up from coarsening), and some of the cell/face numbers are not used (if I remember correctly). But none of these should be a problem for forming the exchange pattern of a matrix.
It looks like something goes wrong in at least the computation of send_targets
and entries_to_be_sent
in certain cases.
I do see the bug now: for local refinement ghost_targets
and import_targets
do not necessarily contain the same processors any more.
For the one case where I have a problem, rank 0 has a completely garbled up communication setup:
entries_to_be_sent:
23 (gl: 23) <-> 23 (gl: 23)
24 (gl: 24) <-> 24 (gl: 24)
25 (gl: 25) <-> 25 (gl: 25)
27 (gl: 27) <-> 27 (gl: 27)
27 (gl: 27) <-> 770 (gl: 770)
27 (gl: 27) <-> 771 (gl: 773)
30 (gl: 30) <-> 30 (gl: 30)
30 (gl: 30) <-> 769 (gl: 769)
30 (gl: 30) <-> 770 (gl: 770)
30 (gl: 30) <-> 771 (gl: 773)
30 (gl: 30) <-> 787 (gl: 809)
32 (gl: 32) <-> 32 (gl: 32)
32 (gl: 32) <-> 770 (gl: 770)
36 (gl: 36) <-> 36 (gl: 36)
36 (gl: 36) <-> 771 (gl: 773)
36 (gl: 36) <-> 787 (gl: 809)
37 (gl: 37) <-> 37 (gl: 37)
46 (gl: 46) <-> 46 (gl: 46)
51 (gl: 51) <-> 51 (gl: 51)
52 (gl: 52) <-> 52 (gl: 52)
53 (gl: 53) <-> 53 (gl: 53)
53 (gl: 53) <-> 769 (gl: 769)
53 (gl: 53) <-> 787 (gl: 809)
53 (gl: 53) <-> 789 (gl: 813)
54 (gl: 54) <-> 54 (gl: 54)
55 (gl: 55) <-> 55 (gl: 55)
55 (gl: 55) <-> 769 (gl: 769)
55 (gl: 55) <-> 775 (gl: 781)
55 (gl: 55) <-> 789 (gl: 813)
57 (gl: 57) <-> 57 (gl: 57)
57 (gl: 57) <-> 769 (gl: 769)
57 (gl: 57) <-> 787 (gl: 809)
58 (gl: 58) <-> 58 (gl: 58)
59 (gl: 59) <-> 59 (gl: 59)
62 (gl: 62) <-> 62 (gl: 62)
63 (gl: 63) <-> 63 (gl: 63)
64 (gl: 64) <-> 64 (gl: 64)
65 (gl: 65) <-> 65 (gl: 65)
66 (gl: 66) <-> 66 (gl: 66)
67 (gl: 67) <-> 67 (gl: 67)
68 (gl: 68) <-> 68 (gl: 68)
68 (gl: 68) <-> 775 (gl: 781)
68 (gl: 68) <-> 789 (gl: 813)
68 (gl: 68) <-> 790 (gl: 814)
69 (gl: 69) <-> 69 (gl: 69)
70 (gl: 70) <-> 70 (gl: 70)
70 (gl: 70) <-> 775 (gl: 781)
70 (gl: 70) <-> 776 (gl: 782)
70 (gl: 70) <-> 790 (gl: 814)
71 (gl: 71) <-> 71 (gl: 71)
71 (gl: 71) <-> 786 (gl: 798)
71 (gl: 71) <-> 788 (gl: 810)
74 (gl: 74) <-> 74 (gl: 74)
75 (gl: 75) <-> 75 (gl: 75)
76 (gl: 76) <-> 76 (gl: 76)
77 (gl: 77) <-> 77 (gl: 77)
78 (gl: 78) <-> 78 (gl: 78)
79 (gl: 79) <-> 79 (gl: 79)
79 (gl: 79) <-> 782 (gl: 791)
79 (gl: 79) <-> 788 (gl: 810)
80 (gl: 80) <-> 80 (gl: 80)
81 (gl: 81) <-> 81 (gl: 81)
82 (gl: 82) <-> 82 (gl: 82)
82 (gl: 82) <-> 783 (gl: 793)
82 (gl: 82) <-> 784 (gl: 795)
82 (gl: 82) <-> 793 (gl: 866)
83 (gl: 83) <-> 83 (gl: 83)
83 (gl: 83) <-> 783 (gl: 793)
83 (gl: 83) <-> 784 (gl: 795)
83 (gl: 83) <-> 785 (gl: 797)
84 (gl: 84) <-> 84 (gl: 84)
85 (gl: 85) <-> 85 (gl: 85)
88 (gl: 88) <-> 88 (gl: 88)
88 (gl: 88) <-> 776 (gl: 782)
88 (gl: 88) <-> 782 (gl: 791)
88 (gl: 88) <-> 786 (gl: 798)
88 (gl: 88) <-> 788 (gl: 810)
88 (gl: 88) <-> 790 (gl: 814)
88 (gl: 88) <-> 791 (gl: 815)
89 (gl: 89) <-> 89 (gl: 89)
89 (gl: 89) <-> 781 (gl: 790)
89 (gl: 89) <-> 786 (gl: 798)
89 (gl: 89) <-> 788 (gl: 810)
89 (gl: 89) <-> 792 (gl: 865)
90 (gl: 90) <-> 90 (gl: 90)
90 (gl: 90) <-> 777 (gl: 784)
90 (gl: 90) <-> 781 (gl: 790)
90 (gl: 90) <-> 782 (gl: 791)
90 (gl: 90) <-> 783 (gl: 793)
90 (gl: 90) <-> 785 (gl: 797)
90 (gl: 90) <-> 786 (gl: 798)
90 (gl: 90) <-> 792 (gl: 865)
92 (gl: 92) <-> 92 (gl: 92)
92 (gl: 92) <-> 784 (gl: 795)
92 (gl: 92) <-> 793 (gl: 866)
100 (gl: 100) <-> 100 (gl: 100)
105 (gl: 105) <-> 105 (gl: 105)
send_targets:
rank = 1 with offset = 97
rank = 2 with offset = 97 # sending nothing
rank = 3 with offset = 97 # sending nothing
rank = 4 with offset = 97 # sending nothing
rank = 6 with offset = 97 # sending nothing
While on the receive side it looks like this: (error messages not necessarily in order)
Found dof 39 in the ghost range on rank 3
receive_targets:
rank = 0 with offset = 41
rank = 1 with offset = 77
rank = 2 with offset = 288
rank = 4 with offset = 326
rank = 6 with offset = 371
Found dof 39 in the ghost range on rank 6
receive_targets:
rank = 0 with offset = 5
rank = 3 with offset = 55
rank = 4 with offset = 322
rank = 5 with offset = 617
rank = 7 with offset = 724
Found dof 39 in the ghost range on rank 4
receive_targets:
rank = 0 with offset = 108
rank = 3 with offset = 157
rank = 5 with offset = 352
rank = 6 with offset = 603
rank = 7 with offset = 605
m_ij matrix is not symmetric: 0.0108507 <-> 0
i: 39 (gl: 2268) <-> j: 718 (gl: 39)
[0, 709) [709, 868)
ghost row: 0 ; 0 ; 0 ;
m_ij matrix is not symmetric: 0.00830756 <-> 0
i: 37 (gl: 2975) <-> j: 778 (gl: 39)
[0, 750) [750, 976)
ghost row: 0 ; 0 ; 0 ; 0 ; 0 ;
m_ij matrix is not symmetric: 0.00271267 <-> 0
i: 168 (gl: 4587) <-> j: 684 (gl: 39)
[0, 684) [684, 979)
ghost row: 0 ; 0 ;
ghost_targets and import_targets do not necessarily contain the same processors any more
Yes, this is indeed the case for the vector partitioner (but should get symmetric once we've established our communication patterns). I'm happy that you found it!
With #135 I hit the following assert on certain locally refined meshes:
My gut feeling was that we still have a bug in the
update_ghost_rows()
function of ourSparseMatrixSIMD
.Interesting observation: local refinement only works. I trigger this assert with mixed refinement / coarsening.
@kronbichler ping