Homa throughput and RTT is not as good as TCP?

Hi,

I have two debian servers running v5.10.70. With trivial changes I built HOMA (linux_5.4.80) successfully. Running a few simple tests and I'm a bit surprised that HOMA performance is not as good as TCP, neither throughput nor RTT:

Server side command: ./cp_node server --protocol homa/tcp Client side command: ./cp_node client --ports 1 --protocol homa --workload 1000 1659663158.268025572 Clients: 8.48 Kops/sec, 0.07 Gbps, RTT (us) P50 121.11 P99 123.87 P99.9 183.79, avg. length 1000.0 bytes ./cp_node client --ports 1 --protocol tcp --workload 1000 1659663180.842814970 Clients: 9.99 Kops/sec, 0.08 Gbps, RTT (us) P50 104.64 P99 107.49 P99.9 184.44, avg. length 1000.0 bytes

./cp_node client --ports 1 --protocol homa --workload 1000 --client-max 100 1659663463.048425659 Clients: 70.92 Kops/sec, 0.57 Gbps, RTT (us) P50 861.38 P99 12590.59 P99.9 12851.42, avg. length 1000.0 bytes ./cp_node client --ports 1 --protocol tcp --workload 1000 --client-max 100 1659663434.857581058 Clients: 115.93 Kops/sec, 0.93 Gbps, RTT (us) P50 838.62 P99 1004.78 P99.9 1036.63, avg. length 1000.0 bytes

I'm using 1Gbps ethernet port and there is a switch between the 2 servers. QoS is not configured at switch side.

Also I tried the "lo" interface: ./cp_node client --ports 1 --protocol homa --workload 100 1659660550.276439665 Clients: 9.40 Kops/sec, 0.01 Gbps, RTT (us) P50 22.92 P99 28.06 P99.9 10352.63, avg. length 100.0 bytes ./cp_node client --ports 1 --protocol homa --workload 100 1659660616.124633889 Clients: 51.79 Kops/sec, 0.04 Gbps, RTT (us) P50 16.88 P99 18.23 P99.9 24.29, avg. length 100.0 bytes

Just wondering, did I do the test wrongly? Or is HOMA designed to be better in specific scenarios in data centers and here I'm not expose any of those scenarios? I was expecting a better performance from HOMA.

Appreciate if you could share your thoughts on this. Thanks.

Sorry for my slow response; I was on vacation last week.

I don't see anything wrong with the way you ran the tests, but Homa's performance is definitely worse than I would expect, especially for the first test. I haven't tried Homa on a 1 Gbps network (that's pretty slow for a modern datacenter), but even so, I'd expect better performance than that.

If you're willing to gather some timetraces for me, I'll take a look to see if I can spot anything obvious. The way to do this is to start the first test running, then on the client machine invoke the command "ttprint.py > client.tt" and on the server machine invoke the command "ttprint.py > server.tt" (ttprint.py is in the util subdirectory). Then send me the two timetraces. If you could do the same thing for the last test (which for some reason is a lot faster than the first), I might be able to get some information by comparing them (for some reason, 100B messages are much faster than 1000B messages; they should be about the same).

-John-

On Wed, Aug 10, 2022 at 3:39 AM jiangyan80122 @.***> wrote:

Hi,

I have two debian servers running v5.10.70. With trivial changes I built HOMA (linux_5.4.80) successfully. Running a few simple tests and I'm a bit surprised that HOMA performance is not as good as TCP, neither throughput nor RTT:

Server side command: ./cp_node server --protocol homa/tcp Client side command: ./cp_node client --ports 1 --protocol homa --workload 1000 1659663158.268025572 Clients: 8.48 Kops/sec, 0.07 Gbps, RTT (us) P50 121.11 P99 123.87 P99.9 183.79, avg. length 1000.0 bytes ./cp_node client --ports 1 --protocol tcp --workload 1000 1659663180.842814970 Clients: 9.99 Kops/sec, 0.08 Gbps, RTT (us) P50 104.64 P99 107.49 P99.9 184.44, avg. length 1000.0 bytes

./cp_node client --ports 1 --protocol homa --workload 1000 --client-max 100 1659663463.048425659 Clients: 70.92 Kops/sec, 0.57 Gbps, RTT (us) P50 861.38 P99 12590.59 P99.9 12851.42, avg. length 1000.0 bytes ./cp_node client --ports 1 --protocol tcp --workload 1000 --client-max 100 1659663434.857581058 Clients: 115.93 Kops/sec, 0.93 Gbps, RTT (us) P50 838.62 P99 1004.78 P99.9 1036.63, avg. length 1000.0 bytes

I'm using 1Gbps ethernet port and there is a switch between the 2 servers. QoS is not configured at switch side.

Also I tried the "lo" interface: ./cp_node client --ports 1 --protocol homa --workload 100 1659660550.276439665 Clients: 9.40 Kops/sec, 0.01 Gbps, RTT (us) P50 22.92 P99 28.06 P99.9 10352.63, avg. length 100.0 bytes ./cp_node client --ports 1 --protocol homa --workload 100 1659660616.124633889 Clients: 51.79 Kops/sec, 0.04 Gbps, RTT (us) P50 16.88 P99 18.23 P99.9 24.29, avg. length 100.0 bytes

Just wondering, did I do the test wrongly? Or is HOMA designed to be better in specific scenarios in data centers and here I'm not expose any of those scenarios? I was expecting a better performance from HOMA.

Appreciate if you could share your thoughts on this. Thanks.

— Reply to this email directly, view it on GitHub https://github.com/PlatformLab/HomaModule/issues/9, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOOUCTE4KZ2XIKZFCQ4RCLVYOBHXANCNFSM56EAL4CA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

thanks John for your kindly reply.

The last test was using loopback interface so NIC was not involved. That's why it was better than first one.

after a bit more digging, I suspect this was due to IRQ and thread affinity on my setup: on my server, all IRQs were handled by core 0. My shell was allowed only on core 0, that means homa.ko was loaded on core 0, and cp_node application was on core 0.

To verify my thought, I tried again with a vagrant box, without any any IRQ/thread affinity, and using loopback interface. This time, HOMA is much better than TCP:

vagrant@bullseye:~/homa/util$ ./cp_node client --ports 1 --protocol homa --workload 1000 1660878427.956468219 Average message length 1.0 KB (expected 1.0KB), rate inf K/sec, expected BW inf Gbps 1660878429.967660373 Clients: 74.71 Kops/sec, 0.60 Gbps, RTT (us) P50 12.24 P99 43.28 P99.9 129.01, avg. length 1000.0 bytes 1660878429.967689749 Lag due to overload: 100.0% 1660878429.967691837 Outstanding client RPCs: 1 1660878430.972530632 Clients: 72.38 Kops/sec, 0.58 Gbps, RTT (us) P50 12.32 P99 42.11 P99.9 133.00, avg. length 1000.0 bytes 1660878430.972563788 Lag due to overload: 100.0% 1660878431.977332868 Clients: 73.24 Kops/sec, 0.59 Gbps, RTT (us) P50 12.30 P99 41.46 P99.9 126.37, avg. length 1000.0 bytes 1660878431.977464918 Lag due to overload: 100.0% 1660878431.977506521 Outstanding client RPCs: 1 1660878432.982079033 Clients: 73.46 Kops/sec, 0.59 Gbps, RTT (us) P50 12.37 P99 41.12 P99.9 135.29, avg. length 1000.0 bytes 1660878432.982205683 Lag due to overload: 100.0% 1660878432.982245530 Outstanding client RPCs: 1 1660878433.986878911 Clients: 75.27 Kops/sec, 0.60 Gbps, RTT (us) P50 12.38 P99 36.69 P99.9 113.57, avg. length 1000.0 bytes 1660878433.986925275 Lag due to overload: 100.0% 1660878433.986927448 Outstanding client RPCs: 1

vagrant@bullseye:~/homa/util$ ./cp_node client --ports 1 --protocol tcp --workload 1000 1660878456.911457404 Average message length 1.0 KB (expected 1.0KB), rate inf K/sec, expected BW inf Gbps 1660878458.915320851 Clients: 13.95 Kops/sec, 0.11 Gbps, RTT (us) P50 62.50 P99 150.53 P99.9 234.81, avg. length 1000.0 bytes 1660878458.915359361 Lag due to overload: 100.0% 1660878459.917071080 Clients: 6.41 Kops/sec, 0.05 Gbps, RTT (us) P50 62.90 P99 155.06 P99.9 562.19, avg. length 1000.0 bytes 1660878459.917094273 Lag due to overload: 100.0% 1660878460.919159769 Clients: 6.40 Kops/sec, 0.05 Gbps, RTT (us) P50 63.38 P99 160.16 P99.9 562.19, avg. length 1000.0 bytes 1660878460.919228837 Lag due to overload: 100.0% 1660878461.924046367 Clients: 13.21 Kops/sec, 0.11 Gbps, RTT (us) P50 64.27 P99 163.22 P99.9 582.40, avg. length 1000.0 bytes 1660878461.924068989 Lag due to overload: 100.0% 1660878462.928652610 Clients: 13.44 Kops/sec, 0.11 Gbps, RTT (us) P50 64.29 P99 163.15 P99.9 562.19, avg. length 1000.0 bytes 1660878462.928675637 Lag due to overload: 100.0%

homa_localhost_trace.zip

Attached the timetraces, just in case you'd like to take a look.

I will close this issue.

Thanks for the additional information; I'm glad to hear that Homa performance improved. However, I'm not confident that measurements made with a loop-back interface will be predictive of performance under more realistic conditions (I haven't done any performance tuning for that case).

-John-

On Thu, Aug 18, 2022 at 8:37 PM jiangyan80122 @.***> wrote:

homa_localhost_trace.zip https://github.com/PlatformLab/HomaModule/files/9379006/homa_localhost_trace.zip

Attached the timetraces, just in case you'd like to take a look.

I will close this issue.

— Reply to this email directly, view it on GitHub https://github.com/PlatformLab/HomaModule/issues/9#issuecomment-1220211477, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOOUCRNL2XEKLVKHIC4OUDVZ36PTANCNFSM56EAL4CA . You are receiving this because you commented.Message ID: @.***>

agree. I'll keep you informed if I see any performance issue on target.

Thanks for the additional information; I'm glad to hear that Homa performance improved. However, I'm not confident that measurements made with a loop-back interface will be predictive of performance under more realistic conditions (I haven't done any performance tuning for that case). … -John- On Thu, Aug 18, 2022 at 8:37 PM jiangyan80122 @.> wrote: homa_localhost_trace.zip https://github.com/PlatformLab/HomaModule/files/9379006/homa_localhost_trace.zip Attached the timetraces, just in case you'd like to take a look. I will close this issue. — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOOUCRNL2XEKLVKHIC4OUDVZ36PTANCNFSM56EAL4CA . You are receiving this because you commented.Message ID: @.>

PlatformLab / HomaModule

Homa throughput and RTT is not as good as TCP? #9