sending messages in batch

[siddontang]

I wrote a simple test for gRPC - send 10 msgs with buffer hint and send one msg which contains 10 msgs in batch.

The proto is:

syntax = "proto3";

package raft;

message Peer {      
    uint64 id = 1;
    uint64 store_id = 2;
    bool is_learner = 3;
}

message RegionEpoch {
    uint64 conf_ver = 1;
    uint64 version = 2;
}

message Heartbeat {
    uint64 to = 1;
    uint64 term = 2;
    uint64 log_term = 3;
    uint64 index = 4;
    uint64 commit = 5;
}

message Message {
    uint64 region_id = 1;
    Peer from_peer = 2;
    Peer to_peer = 3;
    RegionEpoch epch = 4;
    Heartbeat msg = 5;
}

message Messages {
    repeated Message msgs = 1;
}

message Done {
}

service Raft {
    rpc One(stream Message) returns (Done) {}
    rpc Multi(stream Messages) returns (Done) {}
}

The server implementation is very easy - receive all messages and reply one Done msg. The client looks:

fn test_one(num: usize, client: &RaftClient) {
    let t = Instant::now();
    let (mut sink, receiver) = client.one().unwrap();
    for _ in 0..num {
        for _ in 0..9 {
            sink = sink.send((new_msg(), WriteFlags::default().buffer_hint(true)))
                .wait()
                .unwrap();
        }
        sink = sink.send((new_msg(), WriteFlags::default()))
            .wait()
            .unwrap();
    }
    future::poll_fn(|| sink.close()).wait().unwrap();
    receiver.wait().unwrap();
    println!("one time {:?}", t.elapsed())
}

fn test_multi(num: usize, client: &RaftClient) {
    let t = Instant::now();
    let (mut sink, receiver) = client.multi().unwrap();
    for _ in 0..num {
        sink = sink.send((new_msgs(), WriteFlags::default()))
            .wait()
            .unwrap();
    }
    future::poll_fn(|| sink.close()).wait().unwrap();
    receiver.wait().unwrap();
    println!("multi time {:?}", t.elapsed())
}

Then I use num = 100000 for test and get the result:

multi time Duration { secs: 3, nanos: 135709348 }
one time Duration { secs: 18, nanos: 973024439 }

As you can see, using batch can reduce the total time too much, maybe 5 times shorter. So I think it has a big benefit to send msgs in batch, especially for TiKV <-> TiKV.

But we also need to do a benchmark for it, and we must also consider backward compatibility.

[siddontang]

I introduce a new gRPC m_raft which sends the Raft messages in batch, the performance looks good with YCSB:

The first one is without batch, others are with batch but with different grpc-raft-conn-num, as we can see, the performance increased nearly about 20%. (we should find a proper value for grpc-raft-conn-num)

To my surprised, the I/O util also reduces

[c4pt0r]

Looks good! But why the IO util also reduced?