[DocDB] Need for an API endpoint from tserver to determine current list of yb-masters and the leader amongst them.

[nchandrappa]

Jira Link: DB-4295

Description

The process flow for a Multi-node cluster creation using Yugabyted is as follows:

• Install single node yugabyted

  ./bin/yugabyted start

• To add 2nd node to the cluster (assuming its IP is 127.0.0.1), following command is to be used:

  ./bin/yugabyted start --join=127.0.0.1

• Join parameter is mandatory and it should be the IP address of the node where Master leader is running.
• Repeat Step#2 to add additional nodes to the cluster

Even at a later time, when the user expands the cluster, the user will have to know where the current master-leader is and that should be the value provided to the join parameter of the start command.

The ideal user experience is for a new node to simply provide the IP address of any existing node of the cluster and not bother the User to know or determine where the current Master-leader is. In order to do that, we would want a tserver API endpoint that would return the list of Masters and Master-leader IP addresses. If such an API exists, yugabyted will behind the scenes figure out where the Masters are and therefore it’s transparent to the end user who simply wants to add additional nodes to an existing cluster.

[hulien22]

Unfortunately there is no endpoint exposed now for tservers, so we need to add a new endpoint to yb-ts-cli in order to fetch the list of active masters + master leader.

To do so, we'll need to add the required cli commands to ts-cli.cc and pipe that through. To get the actual list of masters from the tserver, this is stored in opts_.master_addresses_ (and gets updated in TabletServer::UpdateMasterAddresses, see https://github.com/yugabyte/yugabyte-db/blob/2a60dba02e13088200ee4a60a255e165c884db70/src/yb/tserver/tablet_server.cc#L277). So fetching and returning this list should work here.

Note, we will also need to do some deduplication of the info in opts_.master_addresses_, as some master nodes may be in there more than once due to the way that it is constructed, and there may also be some hostnames in there as well..

[nchandrappa]

@hbhanawat Currently, /api/v1/masters endpoint doesn't provide support for DNS-enabled YBDB clusters. Endpoint resolves the DNS name into host ip-address, which causes issues in yugabyted when TLS is enabled, as the certificate generated with DNS name doesn't match the host ip-address.

See the output of /api/v1/masters endpoint compared to yb-admin command -

{
   "master_server_and_type":[
      {
         "master_server":"10.151.0.29:7100",
         "is_leader":false
      }
   ]
}

$ ./build/latest/bin/yb-admin --certs_dir_name=/home/nikhil/var/certs --master_addresses=dev-server-nikhil.us-east1-b.c.yugabyte.internal list_all_masters
Master UUID                             RPC Host/Port           State           Role    Broadcast Host/Port 
261af8ec5ece478b9180907235121cb9        dev-server-nikhil.us-east1-b.c.yugabyte.internal:7100   ALIVE          LEADER   dev-server-nikhil.us-east1-b.c.yugabyte.internal:7100

is it possible for API endpoint to use the hostname give in the rpc_bind_addresses?

[robro612]

Hi @nchandrappa,

My name is Rohan Jha, I'm a Masters student at UT Austin taking a graduate Distributed Systems course. For my course project, I'm interested in contributing to Yugabyte by working on this issue.

I'd also appreciate any pointers/additional context you can provide for me to get started.

Thanks!