wait_other_notice doesn't wait for the right thing

[nyh]

ccmlib's "wait_other_notice" option wants to wait until all the nodes of the cluster are fully aware of each other. This is necessary, for example, if you want to send a CL=ALL request to this cluster - and don't want the coordinator to think one of the replicas is not available and fail the request.

In today's CCM implementation, this wait is implemented by the watch_log_for_alive() function in ccmlib/node.py. This function says that node 127.0.0.1 is fully aware of node 127.0.0.2 when it sees in 127.0.0.1's log a message like

gossip - InetAddress 127.0.60.2 is now UP, status = UNKNOWN

The problem is that Scylla's code doesn't really print this message as the last thing of becoming "fully aware" that the new node is up. Not only does it print it in the middle of some function, even worse - the node becoming "up" isn't the end of the story. After a node comes up it is still listed as "joining" (getting ranges, copying data, etc.) and while it is listed as "joining", the other node still won't use it!

So instead of relying on an untrustworthy log message, CCM should use the ScyllaDB REST API - wait until the other node is in http://{self.address()}:10000/gossiper/endpoint/live but not in http://{self.address()}:10000/storage_service/nodes/joining.

I tested this in the dtest that used to fail because of this bug (consistency_test.py::TestConsistency::test_short_read_delete) and it seems now to work flawlessly.

Note that my proposed solution will only work on ScyllaDB, not Cassandra. I think this is acceptable for now (CCM has different source files for Cassandra anyway).

FYI @fruch @asias @kostja

[kostja]

I believe we need to have a dedicated node state after boot, RPC_READY, and reflect it in raft group0. This is even planned now as part of the centralized topology work.