Support configuration to allow picking discrete set of interfaces to turn on sflow in addition to the regex based approach

[yevvievvy]

With switchport regex based approach, it becomes almost impossible to turn sflow on a set of discrete interfaces, without inadvertently turning it on unintended interfaces.

Support to allow list of discrete interfaces other than alongside the current regex based approach, allows selectively choosing interfaces to turn sflow on, especially a useful feature for switches.

Eg switchport = e101-001-1, e101-109-0, e102-123-2 or multiple switchport configurations switchport = e101-001-1, switchport = e101-109-0, etc.,

[sflow]

When the sampling is ingress-only it hardly ever makes sense to enable sFlow on a subset of the switch ports. What is the use-case that you have in mind?

Per-port configuration usually invites more problems than it solves. In any moderate-sized network it is way too easy to end up with unintended gaps in coverage. Leaving it out of the hsflowd config was a deliberate choice.

It hadn't even occurred to me that you could use the regular-expression to do it, but I suppose it would look something like this: ^e101-001-1$|^e101-109-0&|^e102-123-2$

Does that work? Ugly, but maybe it serves to discourage(!)

[yevvievvy]

It works. Thats good for a solution. Bond interface config works too. 👍

Now I have 2 requests: 1, During this test, however, I found that if I apply a new config (say I drop one of the ports from the switchport config) and restart hsflowd, it doesn't trigger clean up of the removed port config from the hardware. Even as I monitored continuously the

unintended port has sflow turned-on in the hardware.

Sflow on member port removed from a (multi-member) bond interface also is left uncleaned.

2, Can a SIGHUP handler (or service hsflowd reload) to reload the config be provided & possibly triggering a disable of the previously programmed sflow on the hardware (for the delta in switchport )?

[sflow]

To support per-interface on/off properly would require some changes. I'm still not persuaded that it is worth the complexity, though. What are the use cases?

If we decide it's worthwhile, then there are several choices: it could be done via hsflowd.conf or maybe via DBUS method calls. It really depends on whether the controller is like Puppet or like Python. What would it look like for you?

If it's done via hsflowd.conf then SIGHUP-triggered reload might be tempting -- except that in practice such sampling-changes changes are very rare. We used to handle SIGHUP but we took that out because of the state-explosion that occurs when you have to worry about every setting transitioning on->off as well as off->on.

[yevvievvy]

To state it precisely - 'Disabling sflow on ports where it was turned-on, on receiving SIGTERM (service hsflowd stop)' is the expectation. This in my understanding is missing & if it can be incorporated would solve the remaining part of my deployment problem.

In terms of usecase, in a distribution layer DC switch, I'm intereseted to only understand traffic flowing in the North - South without having to be bothered about the east-west traffic or vice-versa, being able to pick a discrete set of ports to turn sflow on/off would be a very desirable feature. Would prefer that switches don't have to collect/export traffic on unintended port - to - drop it in the collector tool / analysis. Should you choose to support per-port on/off, dbus is an agreeable mechanism.

[sflow]

Agreed. We should be turning off hardware sampling on graceful exit. I will make that change.

[sflow]

I checked changes into master that should turn off sampling on graceful exit, but I don't have an OS10 switch here to try it on. Please "git pull" and let me know if it really does relinquish the hardware resources cleanly on SIGTERM.

[yevvievvy]

sflow is not turned-off, after "kill -s SIGTERM pidof hsflowd" / service hsflowd stop. More on the official channel

[sflow]

Yes. I was able to test on an OS10 switch today and saw the same thing. There are a couple of changes that need to be made. I will reply here again when there is something new to test.

[sflow]

The hardware sampling is still not turned off on exit, however the mod_dbus now has an interface for listing the switch ports and their speed/sampling/polling settings:

https://github.com/sflow/host-sflow/blob/master/src/Linux/scripts/telemetry.py#L19-L23

[sflow]

The latest, version 2.0.8, will now turn off sampling on graceful (SIGTERM) exit. However it requires a change to /opt/dell/os10/bin/cps_config_sflow to allow args "disable " that may not be in the latest OS10 image yet.

Also, the DBUS API now includes an new interface called "switchport" which will list the ifSpeed, sampling-rate and polling-interval for all hardware switch ports. See: https://github.com/sflow/host-sflow/blob/master/src/Linux/scripts/telemetry.py#L19-L23 (though Python "dbus" module does not seem to be loaded by default).

[yevvievvy]

That's good news. Tried the telemetry.py and was able to see the sample & switchport statistics. Found one missing statistics though...Can you please also add a stats for udp datagram dropped count ?

On graceful exit, please check if disable can be done per direction. IMU, a plain "... disable if-name" would disable sflow in both the direction, which may be an intended effect

[sflow]

Do you mean UDP datagrams dropped on input to hsflowd (raw samples) or on output from hsflowd to the network?

For the input side, there is a "dropped_samples" counter in net.sflow.hsflowd.telemetry which is driven by the sequence number in the raw-sample messages. But I expect you found that.

For the output side, it's not so easy. It's not clear that Linux will return EAGAIN, EWOULDBLOCK or ENOBUFS when we call sendto() on a socket of type SOCK_DGRAM. If there is no socket buffer space left I think the msg is dropped silently(?) Correct me if I have that wrong!

It looks like there are drop counters in /proc/net/udp and /proc/net/udp6, so I guess we might be able to get them from there. But at the data-rates we are talking about for sFlow it's extremely unlikely that the OS would run out of buffer space. I'm not sure what the defaults are for OS10 for these settings:

sysctl net.core.wmem_default sysctl net.core.wmem_max

but I expect they have room for at least 200 large packets. Since sFlow rarely exports more than 1000 packets/sec from a switch, the OS or network IO would have to be locked up for more than 200mS (an age) before anything would be dropped.

Counting drops on rx at the remote sFlow collector is straightforward, because that can be done with the various sequence numbers that go in the sFlow payload. Since drops are much more likely to happen in transit on on rx, this is where attention is focused.

It occurs to me that sFlow CLI's that report something like "1056 sflow datagrams generated; 1056 sflow datagrams dropped" they are really highlighting a "no route to host" situation. That situation is much more likely. Perhaps it should be handled specially. But with a Linux OS like OS10 it's so easy to log in and run ping,traceroute etc. that it hardly seems worth adding this complexity to hsflowd.

---000---

I don't think the ingress and egress sampling settings can be separated as two different services that can be independently configured and consumed. Whatever is configured, that's what hsflowd has to consume (details on whether the packet was sampled at ingress or egress are not included with the sample, and it's risky to try and infer). So hsflowd has to have exclusive control over the interfaces it is monitoring, and a full disable is consistent with that. Make sense?

[yevvievvy]

Manually logging in & running a ping / traceroute might suit the Base version; however it may not suit Premium version. That said we can try infer drops from reading /proc//net/udp and /proc//net/udp6. Agree with your view on hsflowd having exclusive control over interfaces it is monitoring.

[sflow]

I'm going to close this issue because the original request was met with this commit back in March: https://github.com/sflow/host-sflow/commit/d706ddec28b82b5b89393b80e810a689fe53358a

However please raise new issues for any of the other topics that may need more attention.