USING THE HIBARI "CLUSTER" TOOL

"Cluster" is a simple tool for installing, configuring, and bootstrapping a cluster of Hibari nodes. Before using this tool, you must:

• Download and build a Hibari package from source.
• Ensure that you have the required third party software on your "installer" node and on your target Hibari nodes.
• Set up proper user privileges on the "installer" node and on your target Hibari nodes.

For guidance on these tasks, see the "Getting Started" chapter in your Hibari Application Developer's Guide. (http://hibari.github.com/hibari-doc/hibari-app-developer-guide.en.html)

This README describes how to configure the Cluster tool, and how to use it to install, start, and stop a Hibari cluster. The information in this README is also available in the "Getting Started" chapter in your Hibari Application Developer's Guide.

NOTE: The Cluster tool should meet the needs of most users. However, this tool's "target node" recipe is currently Linux-centric (e.g. useradd, userdel, ...). Patches and contributions for other OS and platforms are welcome. For non-Linux deployments, the Cluster tool is rather simple so installation can be done manually by following the tool's recipe.

==== Configuring the Cluster Installer Tool

The Cluster tool requires some basic configuration information that indicates how you want your Hibari cluster to be set up. You will create a simple text file that specifies your desired configuration, and then later use the file as input when you run the Cluster tool.

It's simplest to create the file in the same working directory in which you downloaded the cluster tool. You can give the file any name that you want; for purposes of these instructions we will use the file name hibari.config.

Below is a sample hibari.config file. The file that you create must include all of these parameters, and the values must be formatted in the same way as in this example (with parentheses and quotation marks as shown). Parameter descriptions follow the example file.

ADMIN_NODES=(dev1 dev2 dev3) BRICK_NODES=(dev1 dev2 dev3) BRICKS_PER_CHAIN=2

ALL_NODES=(dev1 dev2 dev3) ALL_NETA_ADDRS=("10.181.165.230" "10.181.165.231" "10.181.165.232") ALL_NETB_ADDRS=("10.181.165.230" "10.181.165.231" "10.181.165.232") ALL_NETA_BCAST="10.181.165.255" ALL_NETB_BCAST="10.181.165.255" ALL_NETA_TIEBREAKER="10.181.165.1"

ALL_HEART_UDP_PORT="63099" ALL_HEART_XMIT_UDP_PORT="63100"

• ADMIN_NODES
  • Host names of the nodes that will be eligible to run the Hibari Admin Server. For complete information on the Admin Server, see "The Admin Server Application" in the Hibari System Administrator's Guide.
• BRICK_NODES
  • Host names of the nodes that will serve as Hibari storage bricks. Note that in the sample configuration file above there are three storage brick nodes (dev1, dev2, and dev3), and these three nodes are each eligible to run the Admin Server.
• BRICKS_PER_CHAIN
  • Number of bricks per replication chain. For example, with two bricks per chain there will be two copies of the data stored in the chain (one copy on each brick); with three bricks per chain there will be three copies, and so on. For an overview of chain replication, see "Chain Replication for High Availability and Strong Consistency" in the Hibari Application Developer's Guide. For chain replication detail, see the Hibari System Administrator's Guide.
• ALL_NODES
  • This list of all Hibari nodes is the union of ADMIN_NODES and BRICK_NODES.
• ALL_NETA_ADDRS
  • As described in "The Partition Detector Application" in the Hibari System Administrator's guide, the nodes in a multi-node Hibari cluster should be connected by two networks, Network A and Network B, in order to detect and manage network partitions. The ALL_NETA_ADDRS parameter specifies the IP addresses of each Hibari node within Network A, which is the network through which data replication and other Erlang communications will take place. The list of the IP addresses should correspond in order to host names you listed in the ALL_NODES setting.
• ALL_NETB_ADDRS
  • IP addresses of each Hibari node within Network B. Network B is used only for heartbeat broadcasts that help to detect network partitions. The list of the IP addresses should correspond in order to host names you listed in the ALL_NODES setting.
• ALL_NETA_BCAST
  • IP broadcast address for Network A.
• ALL_NETB_BCAST
  • IP broadcast address for Network B.
• ALL_NETA_TIEBREAKER
  • Within Network A, the IP address for the network monitoring application to use as a "tiebreaker" in the event of a partition. If the network monitoring application on a Hibari node determines that Network A is partitioned and Network B is not partitioned, then if the Network A tiebreaker IP address responds to a ping, then the local node is on the "correct" side of the partition. Ideally the tiebreaker should be the address of the Layer 2 switch or Layer 3 router that all Erlang network distribution communications flow through.
• ALL_HEART_UDP_PORT
  • UDP port for heartbeat listener.
• ALL_HEART_XMIT_UDP_PORT
  • UDP port for heartbeat transmitter.

For more detail on network monitoring configuration settings, see the partition-detector's OTP application source file (https://github.com/hibari/partition-detector/raw/master/src/partition_detector.app.src).

CAUTION: In a production setting, Network A and Network B should be physically different networks and network interfaces. However, for testing and development purposes the same physical network can be used for Network A and Network B (as in the sample configuration file above).

As final configuration steps, on each Hibari node:

• Make sure that the /etc/hosts file has entries for all Hibari nodes in the cluster. For example:

10.181.165.230 dev1.your-domain.com dev1 10.181.165.231 dev2.your-domain.com dev2 10.181.165.232 dev3.your-domain.com dev3

• In the system's /etc/sysctl.conf file, set vm.swappiness=0. Swappiness is not desirable for an Erlang VM.

==== Installing Hibari

From your installer node, logged in as the installer user, take these steps to create your Hibari cluster:

1. In the working directory in which you downloaded the Cluster tool and created your cluster configuration file, place a copy of the Hibari tarball package and md5sum file:

$ cd working-directory $ ls -1 clus hibari-X.Y.Z-DIST-ARCH-WORDSIZE-md5sum.txt hibari-X.Y.Z-DIST-ARCH-WORDSIZE.tgz hibari.config $

2. Create the "hibari" user on all Hibari nodes:

$ for i in dev1 dev2 dev3 ; do ./clus/priv/clus.sh -f init hibari $i ; done hibari@dev1 hibari@dev2 hibari@dev3

NOTE: If the "hibari" user already exists on the target nodes, the -f option will forcefully delete and then re-create the "hibari" user.

3. Install the Hibari package on all Hibari nodes, via the newly created "hibari" user:

$ ./clus/priv/clus-hibari.sh -f init hibari hibari.config hibari-X.Y.Z-DIST-ARCH-WORDSIZE.tgz hibari@dev1 hibari@dev2 hibari@dev3

NOTE: By default the Cluster tool installs Hibari into /usr/local/var/lib on the target nodes. If you prefer a different location, before doing the install open the clus.sh script (in your working directory, under /clus/priv/) and edit the CT_HOMEBASEDIR variable.

=== Starting and Stopping a Multi-Node Hibari Cluster

You can use the Cluster installer tool to start and stop your multi-node Hibari cluster, working from the same node from which you managed the installation process. Note that in each of the Hibari commands in this section you'll be referencing the name of the Cluster tool configuration file that you created during the installation procedure.

==== Starting and Bootstrapping the Hibari Cluster

1. Change to the working directory in which you downloaded the Cluster tool, then start Hibari on all Hibari nodes via the "hibari" user:

$ cd working-directory $ ./clus/priv/clus-hibari.sh -f start hibari hibari.config hibari@dev1 hibari@dev2 hibari@dev3

2. If this is the first time you've started Hibari, bootstrap the system via the "hibari" user:

$ ./clus/priv/clus-hibari.sh -f bootstrap hibari hibari.config hibari@dev1 => hibari@dev1 hibari@dev2 hibari@dev3

The Hibari bootstrap process starts Hibari's Admin Server on the first eligible admin node and creates a single table "tab1" serving as Hibari's default table. For information about creating additional tables, see "Creating New Tables" in the Hibari Application Developer's Guide.

NOTE: If bootstrapping fails due to "another_admin_server_running" error, please stop the other Hibari cluster(s) running on the network; or reconfigure the Cluster tool to assign Hibari heartbeat listener ports that are not in use by another Hibari cluster or other applications and then repeat the cluster installation procedure.

==== Verifying the Hibari Cluster

Do these simple checks to verify that Hibari is up and running.

1. Confirm that you can open the "Hibari Web Administration" page:

$ your-favorite-browser http://dev1:23080

2. Confirm that you can successfully ping each of your Hibari nodes:

$ ./clus/priv/clus-hibari.sh -f ping hibari hibari.config hibari@dev1 ... pong hibari@dev2 ... pong hibari@dev3 ... pong

==== Stopping the Hibari Cluster

Stop Hibari on all Hibari nodes via the "hibari" user:

$ cd working-directory $ ./clus/priv/clus-hibari.sh -f stop hibari hibari.config ok ok ok hibari@dev1 hibari@dev2 hibari@dev3