some confusion about lantency

[chenbokai]

QUESTION

"dlwrite.dl.per_channel_bookie_client.192_168_1_41_3181.ADD_ENTRY" : {
  "count" : 707932,
  "max" : 0.039316404,
  "mean" : 7.294639961089494E-4,
  "min" : 2.75805E-4,
  "p50" : 3.984705E-4,
  "p75" : 4.2659500000000004E-4,
  "p95" : 7.601880999999999E-4,
  "p98" : 8.955364600000001E-4,
  "p99" : 0.01701617963000005,
  "p999" : 0.03927668229400001,
  "stddev" : 0.00278056131831563,
  "m15_rate" : 1891.799670421043,
  "m1_rate" : 1635.4133239140433,
  "m5_rate" : 1752.104374396801,
  "mean_rate" : 1818.8154766734172,
  "duration_units" : "seconds",
  "rate_units" : "calls/second"
}
"dlwrite.dl.bookkeeper_client.ADD_ENTRY" : {
  "count" : 707931,
  "max" : 0.013280118,
  "mean" : 5.022563307392996E-4,
  "min" : 3.6949300000000003E-4,
  "p50" : 4.7803300000000005E-4,
  "p75" : 5.090157500000001E-4,
  "p95" : 5.7123375E-4,
  "p98" : 5.973867E-4,
  "p99" : 6.266521E-4,
  "p999" : 0.012919525446000045,
  "stddev" : 4.0124739391973426E-4,
  "m15_rate" : 1891.6915529102328,
  "m1_rate" : 1574.91892387704,
  "m5_rate" : 1726.0401619220734,
  "mean_rate" : 1817.5354915211597,
  "duration_units" : "seconds",
  "rate_units" : "calls/second"
}

Through reading code,i think "per_channel_bookie_client" is part of "bookkeeperclient" about lantency.It confuses me with the lantency shows contrary between both.Can someone explain that? Thank you and forgive my pool English ^^

[sijie]

It is a nice question - the latency of bookkeeper_client is the latency your application would see, the latency of the per_channel_bookie_client is the latency for individual request sent to a bookie. Typically you will see better latency in bookkeeper_client than latency in per_channel_bookie_client; because bookkeeper uses ack quorum for waiting responses from per_channel_bookie_clients.

for example, if you have 3 bookies A, B, C. the latency to A and B are very low - for example, in 2 ms, while C is slow, the latency to C is a bit higher - for example, in 10 ms.

If you write an entry (using ack quorum size == 2), the entry will be completed in 2 ms, because A and B can respond in 2ms. Although, you might still see 10ms latency in per_channel_bookie_client to C.

Hope this explain.

[chenbokai]

Thank you for answer my question.It inspired me that when three bookies run in different processing speed may cause that situation,so i check data. However, bookies processing speed seem like same and they are higher than bookkeeper_client.Here is the data:

"dlwrite.dl.per_channel_bookie_client.192_168_1_41_3181.ADD_ENTRY" : {
  "count" : 707932,
  "max" : 0.039316404,
  "mean" : 7.294639961089494E-4,
  "min" : 2.75805E-4,
  "p50" : 3.984705E-4,
  "p75" : 4.2659500000000004E-4,
  "p95" : 7.601880999999999E-4,
  "p98" : 8.955364600000001E-4,
  "p99" : 0.01701617963000005,
  "p999" : 0.03927668229400001,
  "stddev" : 0.00278056131831563,
  "m15_rate" : 1891.799670421043,
  "m1_rate" : 1635.4133239140433,
  "m5_rate" : 1752.104374396801,
  "mean_rate" : 1818.8154766734172,
  "duration_units" : "seconds",
  "rate_units" : "calls/second"
},
"dlwrite.dl.per_channel_bookie_client.192_168_1_51_3181.ADD_ENTRY" : {
  "count" : 707933,
  "max" : 0.041522023000000005,
  "mean" : 6.320614114785992E-4,
  "min" : 2.9043E-4,
  "p50" : 5.07891E-4,
  "p75" : 5.4467125E-4,
  "p95" : 6.306521E-4,
  "p98" : 8.9673956E-4,
  "p99" : 9.898051300000009E-4,
  "p999" : 0.040984281597000065,
  "stddev" : 0.001659605235734714,
  "m15_rate" : 1891.9023616719758,
  "m1_rate" : 1649.1939141689302,
  "m5_rate" : 1758.0522945453104,
  "mean_rate" : 1819.096539916557,
  "duration_units" : "seconds",
  "rate_units" : "calls/second"
},
"dlwrite.dl.per_channel_bookie_client.192_168_1_61_3181.ADD_ENTRY" : {
  "count" : 707933,
  "max" : 0.038555315,
  "mean" : 6.465415389105059E-4,
  "min" : 2.7362800000000004E-4,
  "p50" : 4.183085E-4,
  "p75" : 4.452785E-4,
  "p95" : 6.577250999999996E-4,
  "p98" : 8.975637999999999E-4,
  "p99" : 0.008250984760000094,
  "p999" : 0.03834831407300003,
  "stddev" : 0.0021840395244698825,
  "m15_rate" : 1891.9064686175013,
  "m1_rate" : 1649.062855162423,
  "m5_rate" : 1757.9963518522115,
  "mean_rate" : 1819.0914454770852,
  "duration_units" : "seconds",
  "rate_units" : "calls/second"
},
"dlwrite.dl.bookkeeper_client.ADD_ENTRY" : {
  "count" : 707931,
  "max" : 0.013280118,
  "mean" : 5.022563307392996E-4,
  "min" : 3.6949300000000003E-4,
  "p50" : 4.7803300000000005E-4,
  "p75" : 5.090157500000001E-4,
  "p95" : 5.7123375E-4,
  "p98" : 5.973867E-4,
  "p99" : 6.266521E-4,
  "p999" : 0.012919525446000045,
  "stddev" : 4.0124739391973426E-4,
  "m15_rate" : 1891.6915529102328,
  "m1_rate" : 1574.91892387704,
  "m5_rate" : 1726.0401619220734,
  "mean_rate" : 1817.5354915211597,
  "duration_units" : "seconds",
  "rate_units" : "calls/second"
},

But i think it maybe a rough conclusion through judging by mean rate. Is there possible that situation: A(below average) B(below average) C(above average);then A(below average) B(above average) C(below average) and in turn. So we can't compare both mean rate because bookkeeper_client is always lower than per_channel_bookie_client.

By the way,will there be any performance improvements in the future? I am very much looking forward to seeing it!

[sijie]

yes it is exactly the case what you just described.

yes, in the current master, we are leveraging bookkeeper 4.5.0, expect to have better memory management and hence better performance.