`SparsityPattern`: Antisymmetry assert for c_ij matrix triggers on locally refined and coarsened meshes

[tamiko]

With #135 I hit the following assert on certain locally refined meshes:

An error occurred in line <1052> of file <source/offline_data.template.h> in function
    void ryujin::OfflineData<2>::assemble() [dim = 2, Number = double]
Additional information: 
    c_ij matrix is not anti-symmetric: -0.0520833 -0.0520833 <-> 0 0

My gut feeling was that we still have a bug in the update_ghost_rows() function of our SparseMatrixSIMD.

Interesting observation: local refinement only works. I trigger this assert with mixed refinement / coarsening.

@kronbichler ping

[kronbichler]

Mixed refinement/coarsening should, in my opinion, not result in any significant difference compared to doing local refinement only. It obvious does lead to different numberings of vertices, the cells are traversed differently (because new cells might get inserted both at the end or in the middle when free space comes up from coarsening), and some of the cell/face numbers are not used (if I remember correctly). But none of these should be a problem for forming the exchange pattern of a matrix.

[tamiko]

It looks like something goes wrong in at least the computation of send_targets and entries_to_be_sent in certain cases.

I do see the bug now: for local refinement ghost_targets and import_targets do not necessarily contain the same processors any more.

[tamiko]

For the one case where I have a problem, rank 0 has a completely garbled up communication setup:

entries_to_be_sent:
23 (gl: 23) <-> 23 (gl: 23)
24 (gl: 24) <-> 24 (gl: 24)
25 (gl: 25) <-> 25 (gl: 25)
27 (gl: 27) <-> 27 (gl: 27)
27 (gl: 27) <-> 770 (gl: 770)
27 (gl: 27) <-> 771 (gl: 773)
30 (gl: 30) <-> 30 (gl: 30)
30 (gl: 30) <-> 769 (gl: 769)
30 (gl: 30) <-> 770 (gl: 770)
30 (gl: 30) <-> 771 (gl: 773)
30 (gl: 30) <-> 787 (gl: 809)
32 (gl: 32) <-> 32 (gl: 32)
32 (gl: 32) <-> 770 (gl: 770)
36 (gl: 36) <-> 36 (gl: 36)
36 (gl: 36) <-> 771 (gl: 773)
36 (gl: 36) <-> 787 (gl: 809)
37 (gl: 37) <-> 37 (gl: 37)
46 (gl: 46) <-> 46 (gl: 46)
51 (gl: 51) <-> 51 (gl: 51)
52 (gl: 52) <-> 52 (gl: 52)
53 (gl: 53) <-> 53 (gl: 53)
53 (gl: 53) <-> 769 (gl: 769)
53 (gl: 53) <-> 787 (gl: 809)
53 (gl: 53) <-> 789 (gl: 813)
54 (gl: 54) <-> 54 (gl: 54)
55 (gl: 55) <-> 55 (gl: 55)
55 (gl: 55) <-> 769 (gl: 769)
55 (gl: 55) <-> 775 (gl: 781)
55 (gl: 55) <-> 789 (gl: 813)
57 (gl: 57) <-> 57 (gl: 57)
57 (gl: 57) <-> 769 (gl: 769)
57 (gl: 57) <-> 787 (gl: 809)
58 (gl: 58) <-> 58 (gl: 58)
59 (gl: 59) <-> 59 (gl: 59)
62 (gl: 62) <-> 62 (gl: 62)
63 (gl: 63) <-> 63 (gl: 63)
64 (gl: 64) <-> 64 (gl: 64)
65 (gl: 65) <-> 65 (gl: 65)
66 (gl: 66) <-> 66 (gl: 66)
67 (gl: 67) <-> 67 (gl: 67)
68 (gl: 68) <-> 68 (gl: 68)
68 (gl: 68) <-> 775 (gl: 781)
68 (gl: 68) <-> 789 (gl: 813)
68 (gl: 68) <-> 790 (gl: 814)
69 (gl: 69) <-> 69 (gl: 69)
70 (gl: 70) <-> 70 (gl: 70)
70 (gl: 70) <-> 775 (gl: 781)
70 (gl: 70) <-> 776 (gl: 782)
70 (gl: 70) <-> 790 (gl: 814)
71 (gl: 71) <-> 71 (gl: 71)
71 (gl: 71) <-> 786 (gl: 798)
71 (gl: 71) <-> 788 (gl: 810)
74 (gl: 74) <-> 74 (gl: 74)
75 (gl: 75) <-> 75 (gl: 75)
76 (gl: 76) <-> 76 (gl: 76)
77 (gl: 77) <-> 77 (gl: 77)
78 (gl: 78) <-> 78 (gl: 78)
79 (gl: 79) <-> 79 (gl: 79)
79 (gl: 79) <-> 782 (gl: 791)
79 (gl: 79) <-> 788 (gl: 810)
80 (gl: 80) <-> 80 (gl: 80)
81 (gl: 81) <-> 81 (gl: 81)
82 (gl: 82) <-> 82 (gl: 82)
82 (gl: 82) <-> 783 (gl: 793)
82 (gl: 82) <-> 784 (gl: 795)
82 (gl: 82) <-> 793 (gl: 866)
83 (gl: 83) <-> 83 (gl: 83)
83 (gl: 83) <-> 783 (gl: 793)
83 (gl: 83) <-> 784 (gl: 795)
83 (gl: 83) <-> 785 (gl: 797)
84 (gl: 84) <-> 84 (gl: 84)
85 (gl: 85) <-> 85 (gl: 85)
88 (gl: 88) <-> 88 (gl: 88)
88 (gl: 88) <-> 776 (gl: 782)
88 (gl: 88) <-> 782 (gl: 791)
88 (gl: 88) <-> 786 (gl: 798)
88 (gl: 88) <-> 788 (gl: 810)
88 (gl: 88) <-> 790 (gl: 814)
88 (gl: 88) <-> 791 (gl: 815)
89 (gl: 89) <-> 89 (gl: 89)
89 (gl: 89) <-> 781 (gl: 790)
89 (gl: 89) <-> 786 (gl: 798)
89 (gl: 89) <-> 788 (gl: 810)
89 (gl: 89) <-> 792 (gl: 865)
90 (gl: 90) <-> 90 (gl: 90)
90 (gl: 90) <-> 777 (gl: 784)
90 (gl: 90) <-> 781 (gl: 790)
90 (gl: 90) <-> 782 (gl: 791)
90 (gl: 90) <-> 783 (gl: 793)
90 (gl: 90) <-> 785 (gl: 797)
90 (gl: 90) <-> 786 (gl: 798)
90 (gl: 90) <-> 792 (gl: 865)
92 (gl: 92) <-> 92 (gl: 92)
92 (gl: 92) <-> 784 (gl: 795)
92 (gl: 92) <-> 793 (gl: 866)
100 (gl: 100) <-> 100 (gl: 100)
105 (gl: 105) <-> 105 (gl: 105)

send_targets:
rank = 1 with offset = 97
rank = 2 with offset = 97 # sending nothing
rank = 3 with offset = 97 # sending nothing
rank = 4 with offset = 97 # sending nothing
rank = 6 with offset = 97 # sending nothing

While on the receive side it looks like this: (error messages not necessarily in order)

Found dof 39 in the ghost range on rank 3
receive_targets:
rank = 0 with offset = 41
rank = 1 with offset = 77
rank = 2 with offset = 288
rank = 4 with offset = 326
rank = 6 with offset = 371

Found dof 39 in the ghost range on rank 6
receive_targets:
rank = 0 with offset = 5
rank = 3 with offset = 55
rank = 4 with offset = 322
rank = 5 with offset = 617
rank = 7 with offset = 724

Found dof 39 in the ghost range on rank 4
receive_targets:
rank = 0 with offset = 108
rank = 3 with offset = 157
rank = 5 with offset = 352
rank = 6 with offset = 603
rank = 7 with offset = 605

    m_ij matrix is not symmetric: 0.0108507 <-> 0
    i: 39 (gl: 2268) <-> j: 718 (gl: 39)
    [0, 709) [709, 868)
    ghost row: 0 ; 0 ; 0 ;

    m_ij matrix is not symmetric: 0.00830756 <-> 0
    i: 37 (gl: 2975) <-> j: 778 (gl: 39)
    [0, 750) [750, 976)
    ghost row: 0 ; 0 ; 0 ; 0 ; 0 ;

    m_ij matrix is not symmetric: 0.00271267 <-> 0
    i: 168 (gl: 4587) <-> j: 684 (gl: 39)
    [0, 684) [684, 979)
    ghost row: 0 ; 0 ;

[kronbichler]

ghost_targets and import_targets do not necessarily contain the same processors any more

Yes, this is indeed the case for the vector partitioner (but should get symmetric once we've established our communication patterns). I'm happy that you found it!