关于ureplicator 非联邦 模式的性能问题

[soraka-hu]

Hi:

I have an urgent question to enquire. My question is about the perfomance of ureplicator. Currently, ureplicators are installed on 3 machines, called master01, master02 and master03 separaterly. The controller roles are started on master01 and master02, and all of the three machines start worker roles.

The speed of copying data within kafka is 3, 000, 000/min, while my goal is around 10, 000, 000/min. If another worker role is added, the speed would be up to 4, 000, 000 per minute.

Since the management cost would be increased when adding more workers here, I wonder if there is any way (setting\optimize) can improve the speed of the task without adding any new worker.

Images below are my settings. The code I used is from branch 0.1.

The command to start the role is:

controller

worker

The jvm optimization of the script to start worker role is:

[yangy0000]

1: what is your TCP round trip latency 2: what is your replication byte rate

Also, you probably need to check the graphite metrics to figure out either consumer or producer is not fast enough

[soraka-hu]

HI: 1.My TCP latency is： 2.what is replication byte rate？ Where can I find it？ 3.how check the graphite metrics to figure? You can write to me personally if it's convenient,Thank you very much. Email: 913337533@qq.com WeChat: huml_1005

[yangy0000]

1/ Please try to change "ling.ms" in producer configuration to 1000 2/ to check the byte rate, you would need to export uReplicator Worker metrics to graphite, but please try 1/ first.

[soraka-hu]

hi: Thank you very mach for you reply. 1.I try to change "ling.ms" in producer configuration to 1000,After a period of time, the speed will increase,for example: After I apply for a topic copy, the replication speed in the first five minutes is slower than that in the next five minutes 2.I want to check the byte rate,but i don't export uReplicator Worker metrics in graphite, Do you have any documentation or configuration or steps?

[yangy0000]

for 0.1 branch, worker metrics are export through jmx, there is an example in the user https://github.com/uber/uReplicator/wiki/uReplicator-User-Guide and the jmx configuration example: https://github.com/uber/uReplicator/blob/master/config/jmxtrans.xml The metrics worth checking are Consumer side: bytes-consumed-rateTotalBlockingQueueSize, ConsumerLag,MirrorMaker-flushLatencyMs,MirrorMaker-commitLatencyMs Producer side: buffer-available-bytes, requests-in-flight,

[soraka-hu]

hi: Thank you very mach for you reply. 1.I tried export uReplicator Worker metrics in graphite ,but i encounter a problem

My configuration is :

[soraka-hu]

hi: Thank you very mach , i try to install graphite,but this cannot be installed offline so i use master code and federation mode, at present, 3 worker ,3 controller,1 manager TPS arrive 230, 000/second. i will colse this issue Thank you very mach