Better auto-selection of $NETIF

[novosirj]

Was planning on adding this to our own wrapper script, but it seems like it would find a better home within parsyncfp proper. I'm only sure of how to implement a portion of it though -- you probably know better than me what the handling of $TARGET really looks like.

In any case, our command line for parsyncfp looks more or less like this:

/usr/local/bin/parsyncfp -v 0 --nowait --altcache '/root/.parsyncfp-backups' -NP 12 --rsyncopts '-a -e "ssh -x -c aes128-gcm@openssh.com -o Compression=no"' --maxload 96 --chunksize=10T --fromlist=$HOMEDIR/$SNAPSHOT.list.allfiles.clean --trimpath=/$FILESYSTEM/.snapshots/$SNAPSHOT --trustme nas1:/zdata/gss/$SNAPSHOT

In this case, for the above command line, rather than just choosing the default route, you can pick the actual interface that will be used for the route this transfer will take:

ip -o route get $(getent hosts nas1 | awk '{print $1}') | perl -nle 'if ( /dev\s+(\S+)/ ) {print $1}'

...though obviously done somewhat differently as you're already running this from within perl. "ip route get" apparently requires an IP address. I'm not sure if there's a fancier way to get one.

This also works on MacOS X, though a little easier as route supports hostnames:

route get nas1 | grep interface | awk '{print $2}'

Of course, this gets slightly more complicated as you have to figure out what to do when $TARGET doesn't have a host name in it, but I'd guess that in all cases, the host name or IP address would appear before a colon. getent hosts is safe for either a host or an IP address, though there might be a smarter way to get that information within perl.

I could try my hand at writing something/submitting a pull request, just might be a little slow.

Also worth mentioning is that route and ifconfig are both deprecated on Linux, in favor of ip. I'm not sure if you did that out of habit, or for compatibility with more operating systems, but something to think about.

[hjmangalam]

Hi Ryan,

thanks for this suggestion.

It looks you want an option that directs pfp to use a specific interface thru which to send bytes rather than the default. I've thought about this, esp for our cluster (since many nodes have multiple IB, 1Gb, 10Gb interfaces), but this is more dependent on the routing rather than the remote host specifically. Default routing generally does a good job of sorting this out.

I did include an option for /monitoring/ different interfaces, since sometimes the data comes IN on a different IF than it goes OUT on (and pfp only monitors bytes OUT). An example of this is reading data on from a network volume that's on 10Gb and then sending it out on IB to a compute host.

Can you give me an example where this would provide a significant improvement over default routing without going full quagga/dynamic routing? Or is full quagga what you want?

harry

On Wednesday, January 15, 2020 10:28:09 PM PST Ryan Novosielski wrote:

Was planning on adding this to our own wrapper script, but it seems like it would find a better home within parsyncfp proper. I'm only sure of how to implement a portion of it though -- you probably know better than me what the handling of $TARGET really looks like. In any case, our command line for parsyncfp looks more or less like this:

/usr/local/bin/parsyncfp -v 0 --nowait --altcache '/root/.parsyncfp-backups' -NP 12 --rsyncopts '-a -e "ssh -x -c aes128-gcm@openssh.com -o Compression=no"' --maxload 96 --chunksize=10T --fromlist=$HOMEDIR/$SNAPSHOT.list.allfiles.clean --trimpath=/$FILESYSTEM/.snapshots/$SNAPSHOT --trustme nas1:/zdata/gss/$SNAPSHOT In this case, for the above command line, rather than just choosing the default route, you can pick the actual interface that will be used for the route this transfer will take:

ip -o route get $(getent hosts nas1 | awk '{print $1}') | perl -nle 'if ( /dev\s+(\S+)/ ) {print $1}'

...though obviously done somewhat differently as you're already running this from within perl. "ip route get" apparently requires an IP address. I'm not sure if there's a fancier way to get one.

This also works on MacOS X, though a little easier as route supports hostnames:

route get nas1 | grep interface | awk '{print $2}'

Of course, this gets slightly more complicated as you have to figure out what to do when $TARGET doesn't have a host name in it, but I'd guess that in all cases, the host name or IP address would appear before a colon. getent hosts is safe for either a host or an IP address, though there might be a smarter way to get that information within perl.

I could try my hand at writing something/submitting a pull request, just might be a little slow.

Also worth mentioning is that route and ifconfig are both deprecated on Linux, in favor of ip. I'm not sure if you did that out of habit, or for compatibility with more operating systems, but something to think about.

Harry Mangalam

[novosirj]

No, that’s not what I meant actually. What I’m referring to is specifically your implementation of automatically selecting $NETIF for monitoring purposes. My local example: your code has it automatically select the interface that has the default route. On our system, and I imagine on other systems or in your case that detects multiple default routes and exits forcing the user to choose, the route to the target may not be the default route. In our case, the default is eno1, but the route that will be used to copy is ens6. Since the system already knows which interface it will use to reach the target, what I’m suggesting is just asking the routing engine (with about the same amount of code) rather than looking for the default route and defaulting to that.

[hjmangalam]

On Thursday, January 16, 2020 8:52:58 AM PST Ryan Novosielski wrote:

No, that’s not what I meant actually. What I’m referring to is specifically your implementation of automatically selecting $NETIF for monitoring purposes. My local example: your code has it automatically select the interface that has the default route. On our system, and I imagine on other systems or in your case that detects multiple default routes and exits forcing the user to choose, the route to the target may not be the default route. However, the system already knows which interface it will use to reach the target. What I’m suggesting is just asking the routing engine (with about the same amount of code) rather than looking for the default route.

Then if I understand your request, pfp already has that option - --interface selects the interface to monitor

--interface|i [s] ...... network interface to monitor (not use; see above).Only SENT bytes are displayed.

[[ Unless changed by '--interface', it assumes and monitors the routable interface.

Or am I still not getting what you mean? hjm

Harry Mangalam

[novosirj]

Still not exactly right. I know that I can manually select a different interface with --interface. My point is that the system knows how it will route traffic to the host whose name ultimately will be part of the $TARGET variable, and that can be asked for by running ip route get <IP>. On our system, this is not the default route/will not use the default interface, so a run of parsyncfp without --interface will monitor an idle interface. I can manually set --interface ens6, but since the routing table knows how it will route traffic to nas1 already, why not automatically select the interface over which the transfer will actually occur, rather than asking for the default route and choosing that interface. This will provide a sensible default in all cases, rather than in many cases monitoring the idle interface that has the default route.

[hjmangalam]

I think I see your problem and your problem is not my problem, or at least not the one that the --interface was designed to address.

I have the reverse problem. On our cluster, we have nodes that have multiple interfaces and some of those interfaces are to different networks (ie, our DMZ, as well as to our local filesystems, and to the generic internet, that are firewalled differently and oddly (and out of our control). I think I set up that option to compensate for dual-homed systems that are routed incorrectly (from our POV).

I'll take a look at your solution (and I admit to being slow to drop 'ifconfig' and 'route').
Our cluster is still running CentOS 6.9 so there's been no reason for me to change it. However, I should (and will) change to the horribly unintuitive 'ip' in an upcoming release.

Thanks. Harry

On Thursday, January 16, 2020 1:38:18 PM PST Ryan Novosielski wrote:

Still not exactly right. I know that I can manually select a different interface with --interface. My point is that the system knows how it will route traffic to the host whose name ultimately will be part of the $TARGET variable, and that can be asked for by running ip route get

. On our system, this is not the default route/will not use the default interface, so a run of parsyncfp without --interface will monitor an idle interface. I can manually set --interface ens6, but since the routing table knows how it will route traffic to nas1 already, why not automatically select the interface over which the transfer will actually occur, rather than asking for the default route and choosing that interface. This will provide a sensible default in all cases, rather than in many cases monitoring the idle interface that has the default route.

Harry Mangalam

[novosirj]

I'm not sure about that; I think our case is similar. In both cases, I'd assume, the --interface switch is needed because parsyncfp chooses the wrong interface to monitor.

Our scenario is that we have a machine that is part of our GPFS cluster. Its default route is to the production network on eno2, which goes to the rest of the campus and to the internet. So if we used parsyncfp to transfer data to most sites -- Google, a central server on our campus, etc.. -- the data would be transferred via the default route/over the default route interface and parsyncfp would show bandwidth updates for the correct interface, since your code gets the interface name for the default route and uses it for, effectively, the value of --interface if not specified:

https://github.com/hjmangalam/parsyncfp/blob/caee3a042c72067622d85ccc86379aca4dd16e1c/parsyncfp#L208

If we copy to host nas1, however, the system will use ens6 for the transfer, because nas1 is on the storage network which is on the same subnet as ens6.

If instead of using the above code, that line read like this (or again, the smarter non-nested-perl example that I'd need to remind myself how to write):

$NETIF = `ip -o route get $(getent hosts $TARGETHOST | awk '{print $1}') | perl -nle 'if ( /dev\s+(\S+)/ ) {print $1}'`

...you would get the actual interface rsync was going to be using, and therefore would be monitoring the actual traffic. Here are a few local examples:

[root@quorum01 bin]# ip -o route get $(getent hosts wasabi.com | awk '{print $1}') | perl -nle 'if ( /dev\s+(\S+)/ ) {print $1}'
eno2
[root@quorum01 bin]# ip -o route get $(getent hosts www.rutgers.edu | awk '{print $1}') | perl -nle 'if ( /dev\s+(\S+)/ ) {print $1}'
eno2
[root@quorum01 bin]# ip -o route get $(getent hosts nas1 | awk '{print $1}') | perl -nle 'if ( /dev\s+(\S+)/ ) {print $1}'
ens6

$TARGETHOST could come from $TARGET, which is defined as:

https://github.com/hjmangalam/parsyncfp/blob/caee3a042c72067622d85ccc86379aca4dd16e1c/parsyncfp#L298

...which on our above example invocation would set $TARGET = "nas1:/zdata/gss/$SNAPSHOT", and therefore one could use something like:

$TARGETHOST = (split(/:/, "$TARGET))[0];

...to get the host portion of the rsync target, for example "www.rutgers.edu" for an example case where $TARGET = "www.rutgers.edu:/tmp/wherever". I suppose one would need to trap for cases where $TARGET is a local PATH and therefore contains no ":".

Is it worth it? I personally think so and am probably willing to write a PR.

[hjmangalam]

OK - back to it after a long break. One reason for not using ip is that it has some different output formats on different platforms:

On our Devonian-age cluster running CentOS 6.9 /hmangala@hpc-login-1-2:~[1]/ 995 $ ip -d -f inet route

hjm@stunted:~ 556 $ ip -d -f inet route

I've added a bit that checks the route to a remote machine and sets the appro interface if it's specified as /[user@]remotehost:/path[2]/ where /'user@'[3]/ is optional, as you described below. Let me know if it works as you expect.

The more difficult problem is discovering which route to use and therefore which interface to monitor for mounts that aren't specified like the above. for example, BeeGFS presents the filesystems like: beegfs_dfs1 623T 72T 551T 12% /dfs1

10.20.20.13:/mmfs1/crsp

So... after more messing around than I want to admit to, I've decided to leave that decision to the user (via the already existing --interface option) if pfp detects a multi- homed machine and it doesn't get an obvious remote host as the target.

thanks for your input. I've just pushed the updates that addressed that as well as a few others. Let me know if they address what you needed. hjm

On Thursday, January 16, 2020 11:48:06 PM PST Ryan Novosielski wrote:

I'm not sure about that; I think our case is similar. In both cases, I'd assume, the --interface switch is needed because parsyncfp chooses the wrong interface to monitor.

Our scenario is that we have a machine that is part of our GPFS cluster. Its default route is to the production network on eno2, which goes to the rest of the campus and to the internet. So if we used parsyncfp to transfer data to most sites -- Google, a central server on our campus, etc.. -- the data would be transferred via the default route/over the default route interface and parsyncfp would show bandwidth updates for the correct interface, since your code gets the interface name for the default route and uses it for, effectively, the value of --interface if not specified:

https://github.com/hjmangalam/parsyncfp/blob/caee3a042c72067622d85ccc8637 9aca4dd16e1c/parsyncfp#L208

If we copy to host nas1, however, the system will use ens6 for the transfer, because nas1 is on the storage network which is on the same subnet as ens6.

If instead of using the above code, that line read like this (or again, the smarter non-nested-perl example that I'd need to remind myself how to write):

$NETIF =ip -o route get $(getent hosts $TARGETHOST | awk '{print $1}') | perl -nle 'if ( /dev\s+(\S+)/ ) {print $1}'`

...you would get the actual interface rsync was going to be using, and therefore would be monitoring the actual traffic. Here are a few local examples:

[root@quorum01 bin]# ip -o route get $(getent hosts wasabi.com | awk
'{print $1}') | perl -nle 'if ( /dev\s+(\S+)/ ) {print $1}' eno2