Cannot create single member cluster

[hypnoce]

Hi,

while trying to make a single node raft quorum work, I ended up in a loophole where the single node A was trying to perform an election that never got a response.

Here is the configuration :

<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
  <TCP
          bind_addr="127.0.0.1"
          bind_port="7800"
          port_range="2"
          enable_diagnostics="true"
  />

  <TCPPING initial_hosts="127.0.0.1[7800]"
           port_range="2"
           max_dynamic_hosts="3"
           async_discovery="true"
  />
  <MERGE3 max_interval="30000"
          min_interval="10000"/>
  <FD_SOCK/>
  <FD_ALL/>
  <VERIFY_SUSPECT timeout="1500"  />
  <BARRIER />
  <pbcast.NAKACK2 xmit_interval="500"
                  xmit_table_num_rows="100"
                  xmit_table_msgs_per_row="2000"
                  xmit_table_max_compaction_time="30000"
                  use_mcast_xmit="false"
                  discard_delivered_msgs="true"/>
  <UNICAST3 xmit_interval="500"
            xmit_table_num_rows="100"
            xmit_table_msgs_per_row="2000"
            xmit_table_max_compaction_time="60000"
            conn_expiry_timeout="0"/>
  <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
                 max_bytes="4M"/>
  <raft.NO_DUPES/>
  <pbcast.GMS print_local_addr="true" join_timeout="2000"/>
  <!--<UFC max_credits="2M"-->
  <!--min_threshold="0.4"/>-->
  <MFC max_credits="2M"
       min_threshold="0.4"/>
  <FRAG2 frag_size="60K"  />
  <RSVP resend_interval="2000" timeout="10000"/>
  <pbcast.STATE_TRANSFER />
  <raft.ELECTION election_min_interval="100" election_max_interval="500"/>
  <raft.RAFT members="A" raft_id="A" resend_interval="1000" />
  <raft.REDIRECT/>
  <raft.CLIENT bind_addr="0.0.0.0" />
</config>

Listenning to the role change event, I only get changed role to Candidate

Is this expected ?

[belaban]

No, this is a bug. Please create an issue and I will fix it. However, is this a realistic scenario?

[hypnoce]

Hi,

I will create a test case for better understanding of the issue.

In some scenarios, like developing a service that uses groups-raft on my developer machine, it's a bit annoying to have to create 3 or 5 instances to get the consensus working. Also, some test deployment of our infrastructure does not require fault tolerancy but needs to be very lightweight. We could abstract the ReplicatedStatemachine and use another one for single node deployment. But it would have been great to be able to use the same one. Last, when adding dynamically nodes to Raft using a service discovery mechanism, the first node that is launched will not be functional.

What do you think ?

Thanks

[valdar]

I think it is a reasonable use case;

Especially for testing purposes, it is common to run just 1 node (it is possible with other raft implementations; etcd as the first example that comes to mind).

Regard, Andrea.

On Fri, Jun 28, 2019 at 8:51 AM François JACQUES notifications@github.com wrote:

Hi,

I will create a test case for better understanding of the issue.

In some scenarios, like developing a service that uses groups-raft on my developer machine, it's a bit annoying to have to create 3 or 5 instances to get the consensus working. Also, some test deployment of our infrastructure does not require fault tolerancy but needs to be very lightweight. We could abstract the ReplicatedStatemachine and use another one for single node deployment. But it would have been great to be able to use the same one. Last, when adding dynamically nodes to Raft using a service discovery mechanism, the first node that is launched will not be functional.

What do you think ?

Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/belaban/jgroups-raft/issues/68?email_source=notifications&email_token=AAAB65BDSB3ZVVU744LBDKLP4WYI7A5CNFSM4H3WF43KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYZHS4Y#issuecomment-506624371, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAB65AWQBHM3DKZ5UX23PTP4WYI7ANCNFSM4H3WF43A .

-- "In a world without walls and fences who needs Windows and Gates?" https://about.me/andrea.tarocchi?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=edit_panel&utm_content=thumb Andrea Tarocchi about.me/andrea.tarocchi https://about.me/andrea.tarocchi?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=edit_panel&utm_content=thumb

[hypnoce]

Hi,

To avoid changing a lot in the current RAFT protocol, I was thinking about adding a SINLGE_MEMBER protocol that will be lower than ELECTION and RAFT and will be used to fake other members. What do you think ?

A very ugly sample that seems to work for my use case.

package org.jgroups.protocols.raft;

import org.jgroups.Event;
import org.jgroups.Header;
import org.jgroups.Message;
import org.jgroups.stack.Protocol;
import org.jgroups.util.ExtendedUUID;

public class SINGLE_MEMBER extends Protocol {

    private RAFT raft;
    private ELECTION election;

    @Override
    public void init() throws Exception {
        raft=findProtocol(RAFT.class);
        election=findProtocol(ELECTION.class);
        super.init();
    }

    private  <T extends Protocol> T findProtocol(Class<T> clazz) {
        for(Protocol p=up_prot; p != null; p=p.getUpProtocol()) {
            if(p.getClass().equals(clazz))
                return (T)p;
        }
        throw new IllegalStateException(clazz.getSimpleName() + " not found above " + this.getClass().getSimpleName());
    }

    @Override
    public Object down(Message msg) {
        if (this.isSingleMember()) {
            Header electionHeader = msg.getHeader(election.getId());
            Header raftHeader = msg.getHeader(raft.getId());
            if (raftHeader != null && electionHeader != null) {
                throw new IllegalStateException("Message contains a raft[" + raftHeader.getClass() + "] and election[" + electionHeader.getClass() + "] headers.");
            }
            if (raftHeader != null) {
                handleRaftHeader(raftHeader);
            } else if (electionHeader != null) {
                handleElectionHeader(electionHeader);
            }
            return null;
        }
        return super.down(msg);
    }

    private void handleElectionHeader(Header header) {
        if (header instanceof VoteRequest) {
            VoteRequest voteRequest = (VoteRequest)header;
            VoteResponse voteResponse = new VoteResponse(voteRequest.term(), true);
            Message message = new Message();
            message.putHeader(election.getId(), voteResponse);
            this.up(message);
        }
    }

    private void handleRaftHeader(Header header) {
        if (header instanceof AppendEntriesRequest) {
            AppendEntriesRequest req = (AppendEntriesRequest) header;
            AppendResult appendResult = new AppendResult(true, req.prev_log_index + 1).commitIndex(raft.commitIndex());
            AppendEntriesResponse resp = new AppendEntriesResponse(req.term(), appendResult);

            /* Generate random sender because the leader has already voted */
            ExtendedUUID randomSender = ExtendedUUID.randomUUID();
            Message message = new Message(raft.leader()).src(randomSender);
            message.putHeader(raft.getId(), resp);
            this.up(message);
        }
    }

    private boolean isSingleMember() {
        return raft.majority() == 1;
    }
}

What do you think ?

[belaban]

I'd prefer handling a single member with the generic algorithm. If you define a single member, then the majority is 1 and a single member should therefore always become leader.

[belaban]

OK, the reason a single member won't form a cluster is that a vote request is never sent to self. When a vote request is received, we determine what happens (e.g. Candidate -> Leader etc). But since we never receive our own vote request, this doesn't happen. I changed this, so vote requests are now sent to all members including self. I also added VoteTest.testSingleMember() to test this.

[belaban]

Released 0.4.3.Final