Not Leader failure response

[halexh]

I have created a simple c app utilizing the dqlite library, called dd. Within it, I have started a dqlite_server and implemented the wire protocol to successfully communicate with it (utilizing a TCP socket). I am trying to take it to the next level, and demonstrate the distributed database portion of it. In order to do that, I have created two instances of this app, that only differ in the following ways:

• Instance A and Instance B utilize completely different directories to store the database data (the first argument passed into dqlite_server_create).
• Instance A utilizes dqlite_server_set_auto_bootstrap() and runs on port 8080 (everything is connecting to one another on the same localhost ip).
• Instance B's dqlite_server runs on port 8082.
• Instance B utilizes dqlite_server_set_auto_join() with the IP/Port of Instance A's dqlite_server (in this case 127.0.0.1:8080).
• Instance B implements the dqlite_server_set_connect_func(), successfully (confirmed via debugging) establishing a connection with Instance A's dqlite_server.
• Instance A performs CREATE and INSERT queries (successfully) on the database, again using the wire protocol.
• Instance B is attempting to perform SELECT queries on the database, only after instance A is up and running, and the results are outlined in the diagram below.

My expectation is that Instance B should communicate with its local dqserver (port 8082) via wire protocol. But any attempt to do so results in a "not leader" response Failure Message. Communicating with instance A's dqserver (port 8080) works perfectly fine. But, the client shouldn't be expected to book keep who the leader is, should it? Maybe I am misunderstanding some basic concepts.

[cole-miller]

But, the client shouldn't be expected to book keep who the leader is, should it? Maybe I am misunderstanding some basic concepts.

Yes, currently this is the client's responsibility. go-dqlite has logic to forward requests to the current leader, and so does the WIP high-level C client library. It's definitely a bit inconvenient if you're writing your own client.

[halexh]

But, the client shouldn't be expected to book keep who the leader is, should it? Maybe I am misunderstanding some basic concepts.

Yes, currently this is the client's responsibility. go-dqlite has logic to forward requests to the current leader, and so does the WIP high-level C client library. It's definitely a bit inconvenient if you're writing your own client.

Speaking of that work in progress C client library. Any update as to when that will be completed?

[freeekanayaka]

But, the client shouldn't be expected to book keep who the leader is, should it? Maybe I am misunderstanding some basic concepts.

Yes, currently this is the client's responsibility. go-dqlite has logic to forward requests to the current leader, and so does the WIP high-level C client library. It's definitely a bit inconvenient if you're writing your own client.

Note that any client of a distributed system must have knowledge of at least some of the nodes that are part of the system. That is necessary because any of those nodes might be down and in that case the client will need to try with another node. That logic might look like:

for (i = 0; i < n; i++) {
    node = all_nodes_the_client_knows_about[i];
    rv = try_connect(node);
    if (rv == 0) {
        break;
   }
}

Since that type of logic must be in place in the client no matter what, it should only be necessary to modify it slightly, like this:

for (i = 0; i < n; i++) {
    node = all_nodes_the_client_knows_about[i];
    rv = try_connect(node);
    if (rv == 0 && node->is_leader) {
        break;
   }
}

All that being said, implementing request forwarding so that the client can connect to any node is surely possible. Note though that it will increase latency (there will be an extra network hop, like a proxy), so all in all perhaps it's better to have the client do a bit of extra work.

[cole-miller]

Speaking of that work in progress C client library. Any update as to when that will be completed?

I apologize for the delay -- I put #525 aside in favor of other work that we saw as higher-priority. I intend to return to it soon.

[halexh]

But, the client shouldn't be expected to book keep who the leader is, should it? Maybe I am misunderstanding some basic concepts.

Yes, currently this is the client's responsibility. go-dqlite has logic to forward requests to the current leader, and so does the WIP high-level C client library. It's definitely a bit inconvenient if you're writing your own client.

Note that any client of a distributed system must have knowledge of at least some of the nodes that are part of the system. That is necessary because any of those nodes might be down and in that case the client will need to try with another node. That logic might look like:

for (i = 0; i < n; i++) {
    node = all_nodes_the_client_knows_about[i];
    rv = try_connect(node);
    if (rv == 0) {
        break;
   }
}

Since that type of logic must be in place in the client no matter what, it should only be necessary to modify it slightly, like this:

for (i = 0; i < n; i++) {
    node = all_nodes_the_client_knows_about[i];
    rv = try_connect(node);
    if (rv == 0 && node->is_leader) {
        break;
   }
}

All that being said, implementing request forwarding so that the client can connect to any node is surely possible. Note though that it will increase latency (there will be an extra network hop, like a proxy), so all in all perhaps it's better to have the client do a bit of extra work.

So I tried doing something similar to this, but what I am seeing is that (going back to my diagram above) instance B's copy of the database does not match instance A's once instance A has closed and handed leadership to instance B.

To give some more detail:

Instance A initializes it's dqlite_server with:

  const char *server_address = "127.0.0.1:8080";
  int err;

  err = dqlite_server_create(aDir, aServer);
  err = dqlite_server_set_address(*aServer, server_address);
  err = dqlite_server_set_auto_bootstrap(*aServer, true);
  err = dqlite_server_start(*aServer);

and Instance B initializes its dqlite_server with:

  const char *addrs[] = {"127.0.0.1:8080"};
  const char *server_address = "127.0.0.1:8082";
  int err;
  int thisClientID = 1234;

  err = dqlite_server_create(aDir, aServer);
  err = dqlite_server_set_address(*aServer, server_address);
  err = dqlite_server_set_auto_join(*aServer, addrs, 1);
  err =
      dqlite_server_set_connect_func(*aServer, endpointConnect, &thisClientID);
  err = dqlite_server_start(*aServer);

I excluded endpointConnect() but it definitely connects to instance A.

• After everything is initialized, instance A successfully creates a database, adds some content via an INSERT, and reads it back confirming it's there via a SELECT.
• At this point, instance A pauses at which point instance B is started and attempts to connect to instance A's dqlite_server, which it does successfully.
• Instance B then successfully reads one of the tables from instance A's dqlite_server via a SELECT command.
• At this point, I manually continue instance A, and it gracefully closes, and calls dqlite_server_handover() before instance A exits.
• I then continue instance B, and it attempts to make another SELECT call, this time failing (because its connection to instance A's dqlite_server no longer exists). It then successfully connects to its own instance of dqlite_server.
• It then is required to send the single world containing the protocol version, the open database message (containing the same database name that instance A created), and finally the SELECT call, which it does successfully.
• Instance B's dqlite_server then responds with a Failure Response message containing "no such table: \<table name that was created by instance A>".

From this, it appears that information applied to the dqlite_server leader is not being propagated to other dqlite_servers.

What am I doing incorrectly?

Edit: Got it working. Found a test case that outlines how this should be done, and followed that.