Client reconnect and connect to new node protocol

yilunzhang commented 5 years ago

Each client needs to be connected to a specific node, to be more specific, predecessor(hash(identifier.pubKey)). This is because in order for a packet to be efficiently delivered to a client, the client has to be connected to the node that is closest to it on the Chord ring, otherwise other nodes don't know how to route the packet.

However, when the node that is connected to a client goes offline, or a node that is closer to the client gets online, the client needs to be connected to another node. To handle these cases properly, the following protocols should be implemented in client:

When the connection (e.g. websocket) with node is closed unexpectedly (user not explicitly call close or equivalent function), client should try to reconnect by getting node first and then connect to it. The retry should be repeated indefinitely until success with some appropriate backoff (e.g. exponentially backoff). Client should NOT try to reconnect directly to the last node connected because node may be offline and topology may have changed.
When client receives WrongNode error msg (code 48001), it should disconnect with the current node and use the msg body as the new node address to connect (as if it is the result of getWsAddr or other equivalent API).

The above protocols have been implemented in JS client.

Implementing these protocols are necessary, otherwise client is not able to connect to node for a long time, and I have received people reporting this bug when using Go client. @trueinsider @RealJohnSmith @termoose

RealJohnSmith commented 5 years ago

Thanks for info, will do in new Java SDK update

RealJohnSmith commented 5 years ago

How would the node send an error message? Normally it sends a JSON reply to a text request like {"Action":"setClient","Desc":"WRONG NODE TO CONNECT","Error":48001,"Result":"89.40.113.208:30002","Version":"1.0.0"} but that is a reply with action=setClient.

Is there a way to simulate/test this behavior? (Connecting to a wrong node in the first place doesn't help, because an error is even before the connection is established (response to setClient)

RealJohnSmith commented 5 years ago

Is there for example way to see which message did not get delivered because of this (and thus, which message needs to be resent using a new node)

yilunzhang commented 5 years ago

In websocket there is no "response", client just get the same message as when sending action=setClient msg on a wrong node.

Message will be delivered even if connected to wrong node (if websocket connection is still live). It only affect receiving msg.

trueinsider commented 5 years ago

Difference in Go with JS was that I was trying to getWsAddr only if there was WRONG_NODE error. I changed it so that it'll do getWsAddr on each reconnect now. Also I was waiting for some time before reconnect, now I'm reconnecting instantly and wait only if connect attempt failed.

termoose commented 5 years ago

Version 0.5.0 of the Elixir client now uses the body of WrongNode messages as the new RPC address on reconnects in accordance with the JavaScript client.

New version can be fetched from https://hex.pm/packages/nkn_client

nknorg / nkn-protocols

Client reconnect and connect to new node protocol #4