Open kbr-scylla opened 4 months ago
Possibly related: 2d9e78b09af14c3a2acbd3af7c8910e514895d36 (cc @piodul )
Possibly related: 2d9e78b (cc @piodul )
I don't think the commit causes this issue. This looks like a different kind of race, but in the same area of gossiper (node-specific FD loop marking as down vs. gossiper exchange marking as up).
In any case, the way the test is written, there could be another race, and that other race is something that inherently cannot be fixed.
Namely, the test may start removenode (and enter repair stage) before the killed node is even marked as DOWN.
Then the killed node is marked as DOWN in the middle of repair. And even if it gets later marked as UP (so the markings happen in correct order), repair is interrupted.
So while there does seem to be some kind of race in gossiper, causing DOWN to happen after UP (reversed order of notifications), this scenario will not work in general.
@gleb-cloudius let's store up/down states in raft, please!
@gleb-cloudius let's store up/down states in raft, please!
So your idea to fix a wrong failure detector decision is to make it global?
It's not wrong, it's arriving out of order, it's easiest to fix that by linearizing the state changes with Raft.
It's not wrong, it's arriving out of order, it's easiest to fix that by linearizing the state changes with Raft.
What arriving out of order? Local failure detector decision? Take a lock. You do not need raft to solve a race between two local fibers.
Test by @aleksbykov https://github.com/kbr-scylla/scylladb/commit/563af5fa3c16d0a0be40426c33454d83cc1c22ea
Failed on this phase:
Uploaded logs from last failure:
scylla-3131.log scylla-3132.log scylla-3133.log scylla-3136.log scylla-3137.log scylla-3154.log scylla-3156.log scylla-3168.log topology_experimental_raft.test_crash_coordinator_before_streaming.1.log
In short, this is what happened:
Even though the test contains
before doing
removenode
.Apparently, there is some race inside gossiper failure detection code:
closed_error
, decides to mark it as DOWNIn the above run we saw it on scylla-3132
The issue doesn't look critical.