chord/membership: incorrect successor when joining

        logger.go:130: 2023-01-26T07:58:28.720Z INFO    chord/local_membership.go:55    Successfully joined Chord ring  {"node": 114622756829, "predecessor": 93693217140, "successor": 119592975162}
        logger.go:130: 2023-01-26T07:58:28.720Z INFO    chord/local_membership.go:148   Join completed, joiner has requested to update pointers {"node": 93693217140}
        logger.go:130: 2023-01-26T07:58:28.721Z INFO    chord/local_tasks.go:133    FingerTable entries updated {"node": 93693217140, "fixed": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]}
        logger.go:130: 2023-01-26T07:58:28.721Z INFO    chord/local_membership.go:156   Join completed, joiner has requested to release membership lock {"node": 119592975162}
        logger.go:130: 2023-01-26T07:58:28.721Z INFO    chord/local_membership.go:71    Joining Chord ring  {"node": 644634094637, "via": ""}
        logger.go:130: 2023-01-26T07:58:28.721Z INFO    chord/local_membership.go:100   incoming join request   {"node": 703332983828, "joiner": 644634094637}
        !!! -> logger.go:130: 2023-01-26T07:58:28.722Z  DEBUG   chord/local_chord.go:216    skip transferring keys to predecessor because predecessor left  {"node": 703332983828, "prev": 688551194597, "new": 644634094637}
        logger.go:130: 2023-01-26T07:58:28.722Z INFO    chord/local_tasks.go:133    FingerTable entries updated {"node": 644634094637, "fixed": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42]}
        logger.go:130: 2023-01-26T07:58:28.722Z INFO    chord/local_membership.go:55    Successfully joined Chord ring  {"node": 644634094637, "predecessor": 688551194597, "successor": 703332983828}
        logger.go:130: 2023-01-26T07:58:28.722Z INFO    chord/local_tasks.go:133    FingerTable entries updated {"node": 3930596034382, "fixed": [39]}
        logger.go:130: 2023-01-26T07:58:28.722Z INFO    chord/local_membership.go:148   Join completed, joiner has requested to update pointers {"node": 688551194597}
        logger.go:130: 2023-01-26T07:58:28.723Z INFO    chord/local_chord.go:91 Discovered new predecessor via Notify   {"node": 703332983828, "previous": 644634094637, "predecessor": 688551194597}
        logger.go:130: 2023-01-26T07:58:28.723Z INFO    chord/local_tasks.go:98 Discovered new successors via Stablize  {"node": 329708973517, "successors": [688551194597, 703332983828, 1008538930581, 1012275919287]}

Since we are always calling .Join() on nodes[0], it is possible that its finger table is not updated yet:

node 703332983828 has predecessor 688551194597 and successor (not important)
- INFO chord/local_membership.go:55 Successfully joined Chord ring {"node": 703332983828, "predecessor": 688551194597, "successor": 1008538930581}
node 644634094637 has requested to join via nodes[0], found that 644634094637's successor to be 703332983828
- INFO chord/local_membership.go:100 incoming join request {"node": 703332983828, "joiner": 644634094637}
however 644634094637's successor should really be 688551194597, thus causing this failure
- DEBUG chord/local_chord.go:216 skip transferring keys to predecessor because predecessor left {"node": 703332983828, "prev": 688551194597, "new": 644634094637}

Should add additional check to ensure that the join request is routed to the correct successor in .RequestToJoin()

zllovesuki / specter

chord/membership: incorrect successor when joining #23