transfer throughput of single node

[vkuznet]

We need to find out empirically a node throughput based on hardware specs, i.e. is it better to transfer 10 files at once or 100?

[rishiloyola]

@vkuznet Can you tell me more about this? I guess we can resolve this issue by current tests.

[vkuznet]

Rishi, we need to test how much simultaneous transfers single node can sustain and how it's throughput drops with adding new transfer. To answer this question you need to create N files of equal size and start transfers each of them one at a time, i.e. start one, then another, then another. Each time you can measure how throughput drop if you new transfer is added to the pool. Eventually you need to find how much simultaneous transfer node can sustain.

[rishiloyola]

• I performed this test. One thing I can conclude is if the two requests have shared resources then their transfer rates eventually drops and become extremely low(around 20Mb). In the beginning, I was transferring two same files(shared resource) with the rate of 20 Mb. While processing those two requests, I approved one different kind of request to transfer the data. Suddenly, the transfer rate increased and reached up to the 450 Mb.

• With the adding a new transfer request its throughput is sometimes increasing and sometimes decreasing. I didn't see any specific changes in its throughput.

• Without having the shared resources, it is sending the data with the rate of 400-500Mb.

[vkuznet]

What do you mean by shared resources? Is it the same file you transfer twice? and how do you know when resources are not shared? The rates you give should be normalized to something. Is it 400Mb/sec is good or bad depends on actual network card capacity (which you need to provide).

On 0, Rishi notifications@github.com wrote:

I performed this test. One thing I can conclude is if the two requests have shared resources then their transfer rates eventually drops and become extremely low(around 20Mb).

With the adding a new transfer request its throughput is sometimes increasing and sometimes decreasing. I didn't see any specific changes in its throughput. Without having the shared resources, it is sending the data with the rate of 400-500Mb.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/vkuznet/transfer2go/issues/10#issuecomment-322303457

[rishiloyola]

I am considering the two same files as a shared resources. Not any other quantities.

[rishiloyola]

I have Red Hat, Inc Virtio network device. Which is a para-virtualized driver. So I can calculate speed using iperf like tools. I tried to install it on uibo-cms-02.cr.cnaf.infn.it but it failed. I will do it using pre compiled binary on tomorrow.

Reference: https://serverfault.com/questions/738840/get-link-speed-of-an-virtio-net-network-adapter

[vkuznet]

I would not assume that you can install stuff on CNAF node since network interface may require root access, but you have full control of your VM and you can measure its capacity.

On 0, Rishi notifications@github.com wrote:

I have Red Hat, Inc Virtio network device. Which is a para-virtualized driver. So I can calculate speed using iperf like tools. I tried to install it on uibo-cms-02.cr.cnaf.infn.it but it failed.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/vkuznet/transfer2go/issues/10#issuecomment-322322724

[rishiloyola]

Here is the iperf test result between source-server and main-agent:

------------------------------------------------------------
Client connecting to main-agent.cern.ch, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 188.185.113.130 port 38230 connected with 188.185.100.222 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.67 GBytes  1.44 Gbits/sec

[vkuznet]

and what is throughput of network card 100Mbit/sec, 1Gbit/sec or 10Gbit/sec?

[rishiloyola]

I guess it's 1.44 Gbits/sec. I checked out on both the side(server and client). It is transferring all the data successfully.

[vkuznet]

You don't need to guess, just find out hardware specs via lspci or lshw commands, see https://www.cyberciti.biz/faq/linux-list-network-cards-command/

On 0, Rishi notifications@github.com wrote:

I guess it's 1.44 Gbits/sec. I checked out on both the side(server and client). It is transferring all the data successfully.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/vkuznet/transfer2go/issues/10#issuecomment-322768035

[rishiloyola]

@vkuznet I searched it on the internet but could not find specific information related to throughput. The output of lshw -class network command.

 *-network
       description: Ethernet controller
       product: Virtio network device
       vendor: Red Hat, Inc
       physical id: 3
       bus info: pci@0000:00:03.0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: msix bus_master cap_list rom
       configuration: driver=virtio-pci latency=0
       resources: irq:10 ioport:c0a0(size=32) memory:febd1000-febd1fff memory:fe000000-fe003fff memory:feb80000-febbffff
     *-virtio0
          description: Ethernet interface
          physical id: 0
          bus info: virtio@0
          logical name: eth0
          serial: 02:16:3e:01:b7:d6
          capabilities: ethernet physical
          configuration: broadcast=yes driver=virtio_net driverversion=1.0.0 ip=188.185.113.130 link=yes multicast=yes