Feature request: Round Robin on Available Interfaces

[rapier1]

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

[hjmangalam]

thanks for the suggestion,

Do you mean can it use bonded interfaces? Or if there are multiple interfaces with multiple routes, it can use both? Right now, pfp detects multiple interfaces (for the monitoring) and insists that you specify which one you want to watch. It currently sends data thru the interface that your routing provides as the first path to the target.

I'm currently working on the multihost version, so it can send data thru multiple hosts to (optionally) multiple receive hosts, each with different receive paths). That version should be ready for testing in about a week or so. Is that closer to what you want?

harry

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ .

--

Harry Mangalam

[hjmangalam]

And if it's really a round-robin use, how would you want the bytes to actually be allocated? I'm trying to think of a way that an actual RR send would work, especially with the routing.. Can you elaborate? Or give me an example? h

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ .

--

Harry Mangalam

[rapier1]

Really, I'm just thinking about there being a list of interfaces and each rsync picking an interface from that list when it's instantiated. Nothing on a byte level. Basically, we have a situation in our datacenter where we have a host with 8x100Gb interfaces. All/most of the interfaces are in the same CIDR block. So we were thinking that having parsyncfp cycle through the interfaces would help with throughput (assuming that the storage can keep up).

Also, I want to say how helpful parsyncfp has been. We just moved around 3 or 4 PB of data from our old storage system to a new one. Lustre (of course) was the big drag but parsyncfp worked great. I actually built a front end to it so that users could easily build the parsyncfp command and submit them as slurm jobs. I had to modify parsyncfp a bit - some of those changes might be useful so I'll submit a pull request once I get my version merged with HEAD.

On 3/24/21 11:36 AM, Harry Mangalam wrote:

And if it's really a round-robin use, how would you want the bytes to actually be allocated? I'm trying to think of a way that an actual RR send would work, especially with the routing.. Can you elaborate? Or give me an example? h

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe

https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ .

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805928429, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ.

[hjmangalam]

so you want pfp to handle the detection and use of the rsync flag: --address=ADDRESS bind address for outgoing socket to daemon

This would be on the same host, right?

So when each new chunk is started up, it chooses its interface from the list of available?/given? ones? I'd guess you'd probably want to specify the list so as not to bugger those that are being used for other purposes. So something like:

--roundrobin="192.168.0.4, 192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"

If not that, let me know what you're thinking.. The above doesn't sound too hard. Just a slight mod of the flags to each rsync... altho I could easily be missing something... Tho monitoring all those interfaces would be a nightmare...

hjm

On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier @.***> wrote:

Really, I'm just thinking about there being a list of interfaces and each rsync picking an interface from that list when it's instantiated. Nothing on a byte level. Basically, we have a situation in our datacenter where we have a host with 8x100Gb interfaces. All/most of the interfaces are in the same CIDR block. So we were thinking that having parsyncfp cycle through the interfaces would help with throughput (assuming that the storage can keep up).

Also, I want to say how helpful parsyncfp has been. We just moved around 3 or 4 PB of data from our old storage system to a new one. Lustre (of course) was the big drag but parsyncfp worked great. I actually built a front end to it so that users could easily build the parsyncfp command and submit them as slurm jobs. I had to modify parsyncfp a bit - some of those changes might be useful so I'll submit a pull request once I get my version merged with HEAD.

On 3/24/21 11:36 AM, Harry Mangalam wrote:

And if it's really a round-robin use, how would you want the bytes to actually be allocated? I'm trying to think of a way that an actual RR send would work, especially with the routing.. Can you elaborate? Or give me an example? h

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe

< https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805928429>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805946222, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ .

--

Harry Mangalam

[rapier1]

I believe that is what we are looking for. Monitoring could be a pain but one of my colleagues is looking at that. I'll ask him what his thoughts are.

Chris

On 3/24/21 2:24 PM, Harry Mangalam wrote:

so you want pfp to handle the detection and use of the rsync flag: --address=ADDRESS bind address for outgoing socket to daemon

This would be on the same host, right?

So when each new chunk is started up, it chooses its interface from the list of available?/given? ones? I'd guess you'd probably want to specify the list so as not to bugger those that are being used for other purposes. So something like:

--roundrobin="192.168.0.4, 192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"

If not that, let me know what you're thinking.. The above doesn't sound too hard. Just a slight mod of the flags to each rsync... altho I could easily be missing something... Tho monitoring all those interfaces would be a nightmare...

hjm

On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier @.***> wrote:

Really, I'm just thinking about there being a list of interfaces and each rsync picking an interface from that list when it's instantiated. Nothing on a byte level. Basically, we have a situation in our datacenter where we have a host with 8x100Gb interfaces. All/most of the interfaces are in the same CIDR block. So we were thinking that having parsyncfp cycle through the interfaces would help with throughput (assuming that the storage can keep up).

Also, I want to say how helpful parsyncfp has been. We just moved around 3 or 4 PB of data from our old storage system to a new one. Lustre (of course) was the big drag but parsyncfp worked great. I actually built a front end to it so that users could easily build the parsyncfp command and submit them as slurm jobs. I had to modify parsyncfp a bit - some of those changes might be useful so I'll submit a pull request once I get my version merged with HEAD.

On 3/24/21 11:36 AM, Harry Mangalam wrote:

And if it's really a round-robin use, how would you want the bytes to actually be allocated? I'm trying to think of a way that an actual RR send would work, especially with the routing.. Can you elaborate? Or give me an example? h

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe

<

https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805928429>,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805946222, or unsubscribe

https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ .

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806056021, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66AMHUGLVDPJE5BS4KDTFIU4ZANCNFSM4ZXPJEHQ.

[rapier1]

So I talked to my coworker and he basically hacked getavgnetbw and used a wildcard to use ib* instead of ${NETIF}. That's dirty hack though. We talked about it a bit and it's hard because the interface nomenclature doesn't necessarily work with that outside of infiniband interfaces. Unless you specifically ask the user to enter a regex or wildcard solution that would work.

However, if that can be solved it should be possible to just loop through the results of the cat and aggregate the values of /sys/class/net/[ifname]/statistics/rx_bytes. Or even /sys/class/net/[ifname]/statistics/rx_bytes | paste -sd+ - | bc would give you the total across all interfaces in a single command.

He also said that if you make the --interface able to take a list of interfaces you could just loop over that list. Which would be require collecting the values in an array or hash. Which isn't awful as long as you report aggregate values and not per interface. If that sounds like it might work let me know and I'll see about getting you a patch. Might take some time as I'm floored with other work right now.

Chris

On 3/24/21 2:24 PM, Harry Mangalam wrote:

so you want pfp to handle the detection and use of the rsync flag: --address=ADDRESS bind address for outgoing socket to daemon

This would be on the same host, right?

So when each new chunk is started up, it chooses its interface from the list of available?/given? ones? I'd guess you'd probably want to specify the list so as not to bugger those that are being used for other purposes. So something like:

--roundrobin="192.168.0.4, 192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"

If not that, let me know what you're thinking.. The above doesn't sound too hard. Just a slight mod of the flags to each rsync... altho I could easily be missing something... Tho monitoring all those interfaces would be a nightmare...

hjm

On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier @.***> wrote:

Really, I'm just thinking about there being a list of interfaces and each rsync picking an interface from that list when it's instantiated. Nothing on a byte level. Basically, we have a situation in our datacenter where we have a host with 8x100Gb interfaces. All/most of the interfaces are in the same CIDR block. So we were thinking that having parsyncfp cycle through the interfaces would help with throughput (assuming that the storage can keep up).

Also, I want to say how helpful parsyncfp has been. We just moved around 3 or 4 PB of data from our old storage system to a new one. Lustre (of course) was the big drag but parsyncfp worked great. I actually built a front end to it so that users could easily build the parsyncfp command and submit them as slurm jobs. I had to modify parsyncfp a bit - some of those changes might be useful so I'll submit a pull request once I get my version merged with HEAD.

On 3/24/21 11:36 AM, Harry Mangalam wrote:

And if it's really a round-robin use, how would you want the bytes to actually be allocated? I'm trying to think of a way that an actual RR send would work, especially with the routing.. Can you elaborate? Or give me an example? h

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe

<

https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805928429>,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805946222, or unsubscribe

https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ .

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806056021, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66AMHUGLVDPJE5BS4KDTFIU4ZANCNFSM4ZXPJEHQ.

[hjmangalam]

parsyncfp might be thought of as nothing more than a massive hack of rsync, so more hacks would not be out of line. I've more or less hacked in the changes required to add multiple interfaces in the (multihost) version that I'm working on now (tho the 2 are incompatible) and they should be ready to test within a week. I'll send them out to you when they are, prior to sending it to github.

Monitoring the bandwidth isnt so much the problem - pfp already does that with perfquery (on IB) and other tickles, but organizing the returned output might be tricky. I'll play around with it and see what blows up in my face ;) hjm

On Wed, Mar 24, 2021 at 12:52 PM Chris Rapier @.***> wrote:

So I talked to my coworker and he basically hacked getavgnetbw and used a wildcard to use ib* instead of ${NETIF}. That's dirty hack though. We talked about it a bit and it's hard because the interface nomenclature doesn't necessarily work with that outside of infiniband interfaces. Unless you specifically ask the user to enter a regex or wildcard solution that would work.

However, if that can be solved it should be possible to just loop through the results of the cat and aggregate the values of /sys/class/net/[ifname]/statistics/rx_bytes. Or even /sys/class/net/[ifname]/statistics/rx_bytes | paste -sd+ - | bc would give you the total across all interfaces in a single command.

He also said that if you make the --interface able to take a list of interfaces you could just loop over that list. Which would be require collecting the values in an array or hash. Which isn't awful as long as you report aggregate values and not per interface. If that sounds like it might work let me know and I'll see about getting you a patch. Might take some time as I'm floored with other work right now.

Chris

On 3/24/21 2:24 PM, Harry Mangalam wrote:

so you want pfp to handle the detection and use of the rsync flag: --address=ADDRESS bind address for outgoing socket to daemon

This would be on the same host, right?

So when each new chunk is started up, it chooses its interface from the list of available?/given? ones? I'd guess you'd probably want to specify the list so as not to bugger those that are being used for other purposes. So something like:

--roundrobin="192.168.0.4,

192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"

If not that, let me know what you're thinking.. The above doesn't sound too hard. Just a slight mod of the flags to each rsync... altho I could easily be missing something... Tho monitoring all those interfaces would be a nightmare...

hjm

On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier @.***> wrote:

Really, I'm just thinking about there being a list of interfaces and each rsync picking an interface from that list when it's instantiated. Nothing on a byte level. Basically, we have a situation in our datacenter where we have a host with 8x100Gb interfaces. All/most of the interfaces are in the same CIDR block. So we were thinking that having parsyncfp cycle through the interfaces would help with throughput (assuming that the storage can keep up).

Also, I want to say how helpful parsyncfp has been. We just moved around 3 or 4 PB of data from our old storage system to a new one. Lustre (of course) was the big drag but parsyncfp worked great. I actually built a front end to it so that users could easily build the parsyncfp command and submit them as slurm jobs. I had to modify parsyncfp a bit - some of those changes might be useful so I'll submit a pull request once I get my version merged with HEAD.

On 3/24/21 11:36 AM, Harry Mangalam wrote:

And if it's really a round-robin use, how would you want the bytes to actually be allocated? I'm trying to think of a way that an actual RR send would work, especially with the routing.. Can you elaborate? Or give me an example? h

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe

<

https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805928429 ,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub

< https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805946222>, or unsubscribe

< https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806056021>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAKL66AMHUGLVDPJE5BS4KDTFIU4ZANCNFSM4ZXPJEHQ .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806133646, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3YZFWNDZ6KZWUM6U2Y3TFI7G3ANCNFSM4ZXPJEHQ .

--

Harry Mangalam

[rapier1]

Cool. I came up with something - 1st pass and not tested but for getavgnetbw something like this should work.

my $if_hash;

my @if_list = split(/,/, $NETIF); foreach my $if @.***_list) { $if = trim($if); $if_hash{if}[0] = cat /sys/class/net/$if/statistics/rx_bytes; $if_hash{if}[1] = cat /sys/class/net/$if/statistics/tx_bytes; }

sleep $CHECKPERIOD;

foreach my $if @.***_list) { $if = trim($if); $rx_sum += cat /sys/class/net/$if/statistics/rx_bytes - $if_hash{$if}[0]; $tx_sum += cat /sys/class/net/$if/statistics/tx_bytes - $if_hash{$if}[1]; }

$avgrec = $rx_sum / $CHECKPERIOD; $avgtrans = $tx_sum / $CHECKPERIOD;

On 3/24/21 4:13 PM, Harry Mangalam wrote:

parsyncfp might be thought of as nothing more than a massive hack of rsync, so more hacks would not be out of line. I've more or less hacked in the changes required to add multiple interfaces in the (multihost) version that I'm working on now (tho the 2 are incompatible) and they should be ready to test within a week. I'll send them out to you when they are, prior to sending it to github.

Monitoring the bandwidth isnt so much the problem - pfp already does that with perfquery (on IB) and other tickles, but organizing the returned output might be tricky. I'll play around with it and see what blows up in my face ;) hjm

On Wed, Mar 24, 2021 at 12:52 PM Chris Rapier @.***> wrote:

So I talked to my coworker and he basically hacked getavgnetbw and used a wildcard to use ib* instead of ${NETIF}. That's dirty hack though. We talked about it a bit and it's hard because the interface nomenclature doesn't necessarily work with that outside of infiniband interfaces. Unless you specifically ask the user to enter a regex or wildcard solution that would work.

However, if that can be solved it should be possible to just loop through the results of the cat and aggregate the values of /sys/class/net/[ifname]/statistics/rx_bytes. Or even /sys/class/net/[ifname]/statistics/rx_bytes | paste -sd+ - | bc would give you the total across all interfaces in a single command.

He also said that if you make the --interface able to take a list of interfaces you could just loop over that list. Which would be require collecting the values in an array or hash. Which isn't awful as long as you report aggregate values and not per interface. If that sounds like it might work let me know and I'll see about getting you a patch. Might take some time as I'm floored with other work right now.

Chris

On 3/24/21 2:24 PM, Harry Mangalam wrote:

so you want pfp to handle the detection and use of the rsync flag: --address=ADDRESS bind address for outgoing socket to daemon

This would be on the same host, right?

So when each new chunk is started up, it chooses its interface from the list of available?/given? ones? I'd guess you'd probably want to specify the list so as not to bugger those that are being used for other purposes. So something like:

--roundrobin="192.168.0.4,

192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"

If not that, let me know what you're thinking.. The above doesn't sound too hard. Just a slight mod of the flags to each rsync... altho I could easily be missing something... Tho monitoring all those interfaces would be a nightmare...

hjm

On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier @.***> wrote:

Really, I'm just thinking about there being a list of interfaces and each rsync picking an interface from that list when it's instantiated. Nothing on a byte level. Basically, we have a situation in our datacenter where we have a host with 8x100Gb interfaces. All/most of the interfaces are in the same CIDR block. So we were thinking that having parsyncfp cycle through the interfaces would help with throughput (assuming that the storage can keep up).

Also, I want to say how helpful parsyncfp has been. We just moved around 3 or 4 PB of data from our old storage system to a new one. Lustre (of course) was the big drag but parsyncfp worked great. I actually built a front end to it so that users could easily build the parsyncfp command and submit them as slurm jobs. I had to modify parsyncfp a bit - some of those changes might be useful so I'll submit a pull request once I get my version merged with HEAD.

On 3/24/21 11:36 AM, Harry Mangalam wrote:

And if it's really a round-robin use, how would you want the bytes to actually be allocated? I'm trying to think of a way that an actual RR send would work, especially with the routing.. Can you elaborate? Or give me an example? h

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe

<

https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805928429 ,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub

<

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805946222>,

or unsubscribe

<

https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806056021>,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAKL66AMHUGLVDPJE5BS4KDTFIU4ZANCNFSM4ZXPJEHQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806133646, or unsubscribe

https://github.com/notifications/unsubscribe-auth/AASF3YZFWNDZ6KZWUM6U2Y3TFI7G3ANCNFSM4ZXPJEHQ .

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806156192, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66CBFBCVSI6KWMWLKADTFJBUZANCNFSM4ZXPJEHQ.

[hjmangalam]

Hi Chris, that's a solution close to my heart. :) - it does look quite a bit like what's in there already. It's the cycling thru the different rysncs and (mostly) how to present it in a coherent manner in the interface, but since each rsync will be talking thru it's own interface and can be told to monitor THAT interface, it may already be solved by just modifying the args to rsync. (ie feeding it the --address=xxx.xxx.xxx.xxx option and translating the interface name to the --interface-=ib0/eth0/etc. I would then have to capture and combine all the rsync outputs to provide the summary line at each checkperiod. (ugly but eminently doable..). Let me get the actual roundrobin routine working first.

hjm

On Wed, Mar 24, 2021 at 1:18 PM Chris Rapier @.***> wrote:

Cool. I came up with something - 1st pass and not tested but for getavgnetbw something like this should work.

my $if_hash;

my @if_list = split(/,/, $NETIF); foreach my $if @.***_list) { $if = trim($if); $if_hash{if}[0] = cat /sys/class/net/$if/statistics/rx_bytes; $if_hash{if}[1] = cat /sys/class/net/$if/statistics/tx_bytes; }

sleep $CHECKPERIOD;

foreach my $if @.***_list) { $if = trim($if); $rx_sum += cat /sys/class/net/$if/statistics/rx_bytes - $if_hash{$if}[0]; $tx_sum += cat /sys/class/net/$if/statistics/tx_bytes - $if_hash{$if}[1]; }

$avgrec = $rx_sum / $CHECKPERIOD; $avgtrans = $tx_sum / $CHECKPERIOD;

On 3/24/21 4:13 PM, Harry Mangalam wrote:

parsyncfp might be thought of as nothing more than a massive hack of rsync, so more hacks would not be out of line. I've more or less hacked in the changes required to add multiple interfaces in the (multihost) version that I'm working on now (tho the 2 are incompatible) and they should be ready to test within a week. I'll send them out to you when they are, prior to sending it to github.

Monitoring the bandwidth isnt so much the problem - pfp already does that with perfquery (on IB) and other tickles, but organizing the returned output might be tricky. I'll play around with it and see what blows up in my face ;) hjm

On Wed, Mar 24, 2021 at 12:52 PM Chris Rapier @.***> wrote:

So I talked to my coworker and he basically hacked getavgnetbw and used a wildcard to use ib* instead of ${NETIF}. That's dirty hack though. We talked about it a bit and it's hard because the interface nomenclature doesn't necessarily work with that outside of infiniband interfaces. Unless you specifically ask the user to enter a regex or wildcard solution that would work.

However, if that can be solved it should be possible to just loop through the results of the cat and aggregate the values of /sys/class/net/[ifname]/statistics/rx_bytes. Or even /sys/class/net/[ifname]/statistics/rx_bytes | paste -sd+ - | bc would give you the total across all interfaces in a single command.

He also said that if you make the --interface able to take a list of interfaces you could just loop over that list. Which would be require collecting the values in an array or hash. Which isn't awful as long as you report aggregate values and not per interface. If that sounds like it might work let me know and I'll see about getting you a patch. Might take some time as I'm floored with other work right now.

Chris

On 3/24/21 2:24 PM, Harry Mangalam wrote:

so you want pfp to handle the detection and use of the rsync flag: --address=ADDRESS bind address for outgoing socket to daemon

This would be on the same host, right?

So when each new chunk is started up, it chooses its interface from the list of available?/given? ones? I'd guess you'd probably want to specify the list so as not to bugger those that are being used for other purposes. So something like:

--roundrobin="192.168.0.4,

192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78"

If not that, let me know what you're thinking.. The above doesn't sound too hard. Just a slight mod of the flags to each rsync... altho I could easily be missing something... Tho monitoring all those interfaces would be a nightmare...

hjm

On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier @.***> wrote:

Really, I'm just thinking about there being a list of interfaces and each rsync picking an interface from that list when it's instantiated. Nothing on a byte level. Basically, we have a situation in our datacenter where we have a host with 8x100Gb interfaces. All/most of the interfaces are in the same CIDR block. So we were thinking that having parsyncfp cycle through the interfaces would help with throughput (assuming that the storage can keep up).

Also, I want to say how helpful parsyncfp has been. We just moved around 3 or 4 PB of data from our old storage system to a new one. Lustre (of course) was the big drag but parsyncfp worked great. I actually built a front end to it so that users could easily build the parsyncfp command and submit them as slurm jobs. I had to modify parsyncfp a bit - some of those changes might be useful so I'll submit a pull request once I get my version merged with HEAD.

On 3/24/21 11:36 AM, Harry Mangalam wrote:

And if it's really a round-robin use, how would you want the bytes to actually be allocated? I'm trying to think of a way that an actual RR send would work, especially with the routing.. Can you elaborate? Or give me an example? h

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe

<

https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805928429 ,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub

<

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805946222 ,

or unsubscribe

<

https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806056021 ,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAKL66AMHUGLVDPJE5BS4KDTFIU4ZANCNFSM4ZXPJEHQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub

< https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806133646>, or unsubscribe

< https://github.com/notifications/unsubscribe-auth/AASF3YZFWNDZ6KZWUM6U2Y3TFI7G3ANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806156192>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAKL66CBFBCVSI6KWMWLKADTFJBUZANCNFSM4ZXPJEHQ .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-806159585, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3Y4F5R64M77WWEHFFKTTFJCKPANCNFSM4ZXPJEHQ .

--

Harry Mangalam

[hjmangalam]

Hi Chris, I just put together this interface and it does emit the correct rsync command, but with my dual-homed system (albeit only a 2wifi setup), it really only uses one interface, regardless of the specified one.

My route shows:

$ route

Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default router.asus.com 0.0.0.0 UG 601 0 0 wlx9cefd5fb0bb5 default router.asus.com 0.0.0.0 UG 602 0 0 wlp3s0 link-local 0.0.0.0 255.255.0.0 U 1000 0 0 wlp3s0 192.168.1.0 0.0.0.0 255.255.255.0 U 601 0 0 wlx9cefd5fb0bb5 192.168.1.0 0.0.0.0 255.255.255.0 U 602 0 0 wlp3s0

and the alternating rsync commands generated from pfp are: rsync --address=192.168.1.223 --bwlimit=1000000 -a -s --log-file=/home/hjm/.parsyncfp/rsync-logfile-14.34.52_2021-03-25_16 --files-from=/home/hjm/.parsyncfp/fpcache/f.16 '/home/hjm' bridgi t:/home/hjm/test

and

rsync --address=192.168.1.186 --bwlimit=1000000 -a -s --log-file=/home/hjm/.parsyncfp/rsync-logfile-14.34.52_2021-03-25_17 --files-from=/home/hjm/.parsyncfp/fpcache/f.17 '/home/hjm' bridgit:/home/hjm/test

But the byte streams show only data flowing on one. This is the case whether the rsyncs are started from parsyncfp or via separate rsyncs started from separate shells. Have you been able to send rsync data out simultaneously on different interfaces and if so, let me know how you did it so I can incorporate that approach into pfp.

On Wed, Mar 24, 2021 at 11:23 AM Harry Mangalam @.***> wrote:

so you want pfp to handle the detection and use of the rsync flag: --address=ADDRESS bind address for outgoing socket to daemon

This would be on the same host, right?

So when each new chunk is started up, it chooses its interface from the list of available?/given? ones? I'd guess you'd probably want to specify the list so as not to bugger those that are being used for other purposes. So something like:

--roundrobin="192.168.0.4, 192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78 "

If not that, let me know what you're thinking.. The above doesn't sound too hard. Just a slight mod of the flags to each rsync... altho I could easily be missing something... Tho monitoring all those interfaces would be a nightmare...

hjm

On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier @.***> wrote:

Really, I'm just thinking about there being a list of interfaces and each rsync picking an interface from that list when it's instantiated. Nothing on a byte level. Basically, we have a situation in our datacenter where we have a host with 8x100Gb interfaces. All/most of the interfaces are in the same CIDR block. So we were thinking that having parsyncfp cycle through the interfaces would help with throughput (assuming that the storage can keep up).

Also, I want to say how helpful parsyncfp has been. We just moved around 3 or 4 PB of data from our old storage system to a new one. Lustre (of course) was the big drag but parsyncfp worked great. I actually built a front end to it so that users could easily build the parsyncfp command and submit them as slurm jobs. I had to modify parsyncfp a bit - some of those changes might be useful so I'll submit a pull request once I get my version merged with HEAD.

On 3/24/21 11:36 AM, Harry Mangalam wrote:

And if it's really a round-robin use, how would you want the bytes to actually be allocated? I'm trying to think of a way that an actual RR send would work, especially with the routing.. Can you elaborate? Or give me an example? h

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe

< https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805928429>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805946222, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ .

--

Harry Mangalam

--

Harry Mangalam

[rapier1]

Let me check. Because of working from home (the pandemic edition) I'm not on site to do this myself.

On 3/26/21 12:29 PM, Harry Mangalam wrote:

Hi Chris, I just put together this interface and it does emit the correct rsync command, but with my dual-homed system (albeit only a 2wifi setup), it really only uses one interface, regardless of the specified one.

My route shows:

$ route

Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default router.asus.com 0.0.0.0 UG 601 0 0 wlx9cefd5fb0bb5 default router.asus.com 0.0.0.0 UG 602 0 0 wlp3s0 link-local 0.0.0.0 255.255.0.0 U 1000 0 0 wlp3s0 192.168.1.0 0.0.0.0 255.255.255.0 U 601 0 0 wlx9cefd5fb0bb5 192.168.1.0 0.0.0.0 255.255.255.0 U 602 0 0 wlp3s0

and the alternating rsync commands generated from pfp are: rsync --address=192.168.1.223 --bwlimit=1000000 -a -s --log-file=/home/hjm/.parsyncfp/rsync-logfile-14.34.52_2021-03-25_16 --files-from=/home/hjm/.parsyncfp/fpcache/f.16 '/home/hjm' bridgi t:/home/hjm/test

and

rsync --address=192.168.1.186 --bwlimit=1000000 -a -s --log-file=/home/hjm/.parsyncfp/rsync-logfile-14.34.52_2021-03-25_17 --files-from=/home/hjm/.parsyncfp/fpcache/f.17 '/home/hjm' bridgit:/home/hjm/test

But the byte streams show only data flowing on one. This is the case whether the rsyncs are started from parsyncfp or via separate rsyncs started from separate shells. Have you been able to send rsync data out simultaneously on different interfaces and if so, let me know how you did it so I can incorporate that approach into pfp.

On Wed, Mar 24, 2021 at 11:23 AM Harry Mangalam @.***> wrote:

so you want pfp to handle the detection and use of the rsync flag: --address=ADDRESS bind address for outgoing socket to daemon

This would be on the same host, right?

So when each new chunk is started up, it chooses its interface from the list of available?/given? ones? I'd guess you'd probably want to specify the list so as not to bugger those that are being used for other purposes. So something like:

--roundrobin="192.168.0.4,

192.168.0.5,192.168.0.7,192.168.0.9,192.168.0.24,192.168.0.134,192.168.0.78 "

If not that, let me know what you're thinking.. The above doesn't sound too hard. Just a slight mod of the flags to each rsync... altho I could easily be missing something... Tho monitoring all those interfaces would be a nightmare...

hjm

On Wed, Mar 24, 2021 at 8:57 AM Chris Rapier @.***> wrote:

Really, I'm just thinking about there being a list of interfaces and each rsync picking an interface from that list when it's instantiated. Nothing on a byte level. Basically, we have a situation in our datacenter where we have a host with 8x100Gb interfaces. All/most of the interfaces are in the same CIDR block. So we were thinking that having parsyncfp cycle through the interfaces would help with throughput (assuming that the storage can keep up).

Also, I want to say how helpful parsyncfp has been. We just moved around 3 or 4 PB of data from our old storage system to a new one. Lustre (of course) was the big drag but parsyncfp worked great. I actually built a front end to it so that users could easily build the parsyncfp command and submit them as slurm jobs. I had to modify parsyncfp a bit - some of those changes might be useful so I'll submit a pull request once I get my version merged with HEAD.

On 3/24/21 11:36 AM, Harry Mangalam wrote:

And if it's really a round-robin use, how would you want the bytes to actually be allocated? I'm trying to think of a way that an actual RR send would work, especially with the routing.. Can you elaborate? Or give me an example? h

On Wed, Mar 24, 2021 at 8:20 AM Chris Rapier @.***> wrote:

It seems that parsyncfp won't make use of multiple interfaces on a multihomed host. Before I start looking at doing myself I was wondering if that was on your radar or if that functionality already exists and I'm just missing it.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40, or unsubscribe

<

https://github.com/notifications/unsubscribe-auth/AASF3YYMURBW54M4TSZLKL3TFH7K5ANCNFSM4ZXPJEHQ

.

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805928429>,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAKL66CXWXFQLPLVXUAGZJLTFIBG5ANCNFSM4ZXPJEHQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub

https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-805946222, or unsubscribe

https://github.com/notifications/unsubscribe-auth/AASF3YYE4DIMLCPEMLWWRIDTFIDWDANCNFSM4ZXPJEHQ .

--

Harry Mangalam

--

Harry Mangalam

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-808354659, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66HJSSWW6VKKO6UIDGDTFSY7HANCNFSM4ZXPJEHQ.

[rapier1]

So I poked at things a bit and to be honest it looks like the --address option in rsync is kind of useless. However, using the ssh option -oBinadAddress=ip seems to work as expected (though I'm running into some performance issues which I think is entirely due to my config here at home). This means that the multiple NICs could be used but requires the inclusion of ssh options on the rsync line. Which obviously means that it won't work between mounted filesystems on the same host. So a specific solution instead of a general one. I think I might try to find out who is maintaining rsync and ask about the --address issue.

edit: Thinking about it I bet the --address is only for rsyncs daemons and clients. If you are using the ssh subsystem for transport it will just drop that option instead of passing it along.

[hjmangalam]

Already posted to the rsync list. But feel free to add your voice. I'll check into the ssh solution. That would be useful for many ppl.

On Fri, Mar 26, 2021, 12:59 PM Chris Rapier @.***> wrote:

So I poked at things a bit and to be honest it looks like the --address option in rsync is kind of useless. However, using the ssh option -oBinadAddress=ip seems to work as expected (though I'm running into some performance issues which I think is entirely due to my config here at home). This means that the multiple NICs could be used but requires the inclusion of ssh options on the rsync line. Which obviously means that it won't work between mounted filesystems on the same host. So a specific solution instead of a general one. I think I might try to find out who is maintaining rsync and ask about the --address issue.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-808477038, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3Y2XWSRS4HNJ5F3RUILTFTRRNANCNFSM4ZXPJEHQ .

[hjmangalam]

I'mot seeing the ssh option working here in concert with rsync: ie , from different shells, running thru 2 wlan interfaces:

rsync --rsh="ssh -oBindAddress=192.168.1.223" -av nacs bridgit:~/test and rsync --rsh="ssh -oBindAddress=192.168.1.186" -av tacg bridgit:~/test

both streams are going thru a single interface. I suspect because the arp table sees the 2 interfaces as 1 MAC.

It's odd that the ssh command itself works correctly - ssh will bind to the appropriate interface and the appropriate bytes stream thru the right interface, but asking rsync to use that ssh channel is failing.

What is the format of your rsync commanda that worked as expected, if not similar to the above?

Harry

On Fri, Mar 26, 2021 at 12:59 PM Chris Rapier @.***> wrote:

So I poked at things a bit and to be honest it looks like the --address option in rsync is kind of useless. However, using the ssh option -oBinadAddress=ip seems to work as expected (though I'm running into some performance issues which I think is entirely due to my config here at home). This means that the multiple NICs could be used but requires the inclusion of ssh options on the rsync line. Which obviously means that it won't work between mounted filesystems on the same host. So a specific solution instead of a general one. I think I might try to find out who is maintaining rsync and ask about the --address issue.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-808477038, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3Y2XWSRS4HNJ5F3RUILTFTRRNANCNFSM4ZXPJEHQ .

--

Harry Mangalam

[hjmangalam]

A guy from the rsync list replied to my question about this. His explanation is that if all the interfaces on your multi-homed system are on the same network, then without explicit routing, one interface (the one with the lowest Metric from the route command) will be used. My home net was set up this way, hence the failure to replicate your experience. If the interfaces are on different nets, then multiple interfaces could be used as you describe. I'll send you a trial version of pfp that should work as you describe if the interfaces are on different networks. I'm still working on replicating such a setup where I can test this locally.

And if the interfaces are bonded to stripe data, then you shouldn't need to do any of this, which I suspect you already know.

Harry

On Fri, Mar 26, 2021 at 4:48 PM Harry Mangalam @.***> wrote:

I'mot seeing the ssh option working here in concert with rsync: ie , from different shells, running thru 2 wlan interfaces:

rsync --rsh="ssh -oBindAddress=192.168.1.223" -av nacs bridgit:~/test and rsync --rsh="ssh -oBindAddress=192.168.1.186" -av tacg bridgit:~/test

both streams are going thru a single interface. I suspect because the arp table sees the 2 interfaces as 1 MAC.

It's odd that the ssh command itself works correctly - ssh will bind to the appropriate interface and the appropriate bytes stream thru the right interface, but asking rsync to use that ssh channel is failing.

What is the format of your rsync commanda that worked as expected, if not similar to the above?

Harry

On Fri, Mar 26, 2021 at 12:59 PM Chris Rapier @.***> wrote:

So I poked at things a bit and to be honest it looks like the --address option in rsync is kind of useless. However, using the ssh option -oBinadAddress=ip seems to work as expected (though I'm running into some performance issues which I think is entirely due to my config here at home). This means that the multiple NICs could be used but requires the inclusion of ssh options on the rsync line. Which obviously means that it won't work between mounted filesystems on the same host. So a specific solution instead of a general one. I think I might try to find out who is maintaining rsync and ask about the --address issue.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-808477038, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3Y2XWSRS4HNJ5F3RUILTFTRRNANCNFSM4ZXPJEHQ .

--

Harry Mangalam

--

Harry Mangalam

[hjmangalam]

Hi Chris, I have an interim roundrobin version ready to test. I don't have a setup with multiple physical interfaces to multiple routes that I can modify, so I can send you the version which has some editable choices for you to try. I'll describe that in the email.

If you send me a direct email, I'll send that version. Once it seems to be working, I'll shove it to github for others to try.

It has a lot of debug output enabled by default to make sure it's roundrobining correctly, but is otherwise functional.

Harry

On Mon, Mar 29, 2021 at 8:23 AM Harry Mangalam @.***> wrote:

A guy from the rsync list replied to my question about this. His explanation is that if all the interfaces on your multi-homed system are on the same network, then without explicit routing, one interface (the one with the lowest Metric from the route command) will be used. My home net was set up this way, hence the failure to replicate your experience. If the interfaces are on different nets, then multiple interfaces could be used as you describe. I'll send you a trial version of pfp that should work as you describe if the interfaces are on different networks. I'm still working on replicating such a setup where I can test this locally.

And if the interfaces are bonded to stripe data, then you shouldn't need to do any of this, which I suspect you already know.

Harry

On Fri, Mar 26, 2021 at 4:48 PM Harry Mangalam @.***> wrote:

I'mot seeing the ssh option working here in concert with rsync: ie , from different shells, running thru 2 wlan interfaces:

rsync --rsh="ssh -oBindAddress=192.168.1.223" -av nacs bridgit:~/test and rsync --rsh="ssh -oBindAddress=192.168.1.186" -av tacg bridgit:~/test

both streams are going thru a single interface. I suspect because the arp table sees the 2 interfaces as 1 MAC.

It's odd that the ssh command itself works correctly - ssh will bind to the appropriate interface and the appropriate bytes stream thru the right interface, but asking rsync to use that ssh channel is failing.

What is the format of your rsync commanda that worked as expected, if not similar to the above?

Harry

On Fri, Mar 26, 2021 at 12:59 PM Chris Rapier @.***> wrote:

So I poked at things a bit and to be honest it looks like the --address option in rsync is kind of useless. However, using the ssh option -oBinadAddress=ip seems to work as expected (though I'm running into some performance issues which I think is entirely due to my config here at home). This means that the multiple NICs could be used but requires the inclusion of ssh options on the rsync line. Which obviously means that it won't work between mounted filesystems on the same host. So a specific solution instead of a general one. I think I might try to find out who is maintaining rsync and ask about the --address issue.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hjmangalam/parsyncfp/issues/40#issuecomment-808477038, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASF3Y2XWSRS4HNJ5F3RUILTFTRRNANCNFSM4ZXPJEHQ .

--

Harry Mangalam

--

Harry Mangalam

--

Harry Mangalam