Gauging interest for a tool contribution

[rmanyari]

Hi,

I wrote a script some months ago that is based of Brendan Gregg's tcptop. It essentially categorizes outgoing IPv4 TCP traffic by subnets (the list of subnets is passed as an argument) and reports the aggregates at some interval.

The use case here is to figure out how much cross-zone, cross-region and cloud-to-premise traffic is generated from individual VMs. I'm aware that cloud providers have flow analysis tools but I still find benefit in having something closer to the VM. It opens doors to do more interesting things such as filtering by PID and eventually by UID.

I'd like to know if there is interest from the maintainers of BCC in having this script. I will take the time to follow the contributing scripts guidelines and make it production ready.

Script: https://github.com/rmanyari/bcc-scripts/blob/master/tcpdest.py

[goldshtn]

I think it would be useful!

[brendangregg]

Could you post example output with an explanation of what it is showing? Any submission to bcc/tools needs this in _example.txt files anyway.

Brendan

On Fri, Feb 2, 2018 at 8:12 PM, rmanyari notifications@github.com wrote:

Hi,

I wrote a script some months ago that is based of Brendan Gregg's tcptop. It essentially categorizes outgoing IPv4 TCP traffic by subnets (the list of subnets is passed as an argument) and reports the aggregates at some interval.

The use case here is to figure out how much cross-zone, cross-region and cloud-to-premise traffic is generated from individual VMs. I'm aware that cloud providers have flow analysis tools but I still find benefit in having something closer to the VM. It opens doors to do more interesting things such as filtering by PID and eventually by UID.

I'd like to know if there is interest from the maintainers of BCC in having this script. I will take the time to follow the contributing scripts guidelines and make it production ready.

Script: https://github.com/rmanyari/bcc-scripts/blob/master/tcpdest.py

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/iovisor/bcc/issues/1571, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDNm5XN6tiutDstg9a91bglRN8cE6Cjks5tQ9zDgaJpZM4R4COY .

[rmanyari]

Here I bucket the traffic in 3 categories:

• Local network (192.168.0.0/24)
• github.com (192.30.253.0/24 - I got this from tcpdump, just an approximation for demonstration purposes)
• Everything else

On the left is the category, on the right the number of bytes sent.

root@rodrigo-XPS-13-9360:~/dev/bcc-scripts# ./tcpdest.py -S 192.168.0.0/24,192.30.253.0/24
0.0.0.0/0                 35
0.0.0.0/0               2106
192.30.253.0/24         1813
192.30.253.0/24         2093
0.0.0.0/0               1836
192.30.253.0/24        18068
0.0.0.0/0               5876
192.168.0.0/24             1
0.0.0.0/0               2242
192.168.0.0/24             8
192.168.0.0/24            12
0.0.0.0/0                 35

Output in JSON format:

root@rodrigo-XPS-13-9360:~/dev/bcc-scripts# ./tcpdest.py -S 192.168.0.0/24,192.30.253.0/24 -J
{}
{"192.168.0.0/24": 9.0, "192.30.253.0/24": 35.0, "0.0.0.0/0": 35.0}
{}
{"0.0.0.0/0": 6406.0}
{"0.0.0.0/0": 342.0}
{"192.30.253.0/24": 1813.0, "0.0.0.0/0": 2813.0}
{"0.0.0.0/0": 892.0}
{"192.168.0.0/24": 1.0, "192.30.253.0/24": 20290.0, "0.0.0.0/0": 1839.0}
{"192.168.0.0/24": 14.0}
{}
{"0.0.0.0/0": 3398.0}
{"0.0.0.0/0": 35.0}
{}

[brendangregg]

I've taken a look at the code; some comments:

• It's a good example of subnet processing and summarization in BPF, so I think it could at least should go in bcc examples/tracing for that alone. But it's probably better suited for bcc tools/, with a man page and example file.
• I don't want to encourage people to do tcp send/receive tracing without first exploring other options (like netlink), since the overhead of send/receive tracing can be high. That's what I discussed in http://www.brendangregg.com/blog/2016-10-15/linux-bcc-tcptop.html , and of course netlink has its caveats too. I'd like people to have at least thought about netlink and alternates, to defend why they ended up doing send/receive tracing.
• That it only does tcp_sendmsg() I'd say is a good thing. Easier to maintain, half the overhead.
• I wouldn't call it tcpdest. Maybe tcpsubnet. The man page can explain that it only traces dest.
• Could the default be something other than 0.0.0.0/24? By default (no arguments provided), maybe it'd be nice to trace: 127.0.0.1/32,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
• I'd drop the "-S", since that's its primary function, so the synopsis would be: tcpdest.py [-h] [-v] [-J] [-I INTERVAL] [-s SIZE] [SUBNETS]
• I like the code -- it's neat. It'll need some tweaks to pass the pep8 checks.

Brendan

On Sat, Feb 3, 2018 at 6:34 PM, rmanyari notifications@github.com wrote:

Here I bucket the traffic in 3 categories:

• Local network (192.168.0.0/24)
• github.com (192.30.253.0/24 - I got this from tcpdump, just an approximation for demonstration purposes)
• Everything else

root@rodrigo-XPS-13-9360:~/dev/bcc-scripts# ./tcpdest.py -S 192.168.0.0/24,192.30.253.0/240.0.0.0/0 350.0.0.0/0 2106192.30.253.0/24 1813192.30.253.0/24 20930.0.0.0/0 1836192.30.253.0/24 180680.0.0.0/0 5876192.168.0.0/24 10.0.0.0/0 2242192.168.0.0/24 8192.168.0.0/24 120.0.0.0/0 35

Output in JSON format:

root@rodrigo-XPS-13-9360:~/dev/bcc-scripts# ./tcpdest.py -S 192.168.0.0/24,192.30.253.0/24 -J {} {"192.168.0.0/24": 9.0, "192.30.253.0/24": 35.0, "0.0.0.0/0": 35.0} {} {"0.0.0.0/0": 6406.0} {"0.0.0.0/0": 342.0} {"192.30.253.0/24": 1813.0, "0.0.0.0/0": 2813.0} {"0.0.0.0/0": 892.0} {"192.168.0.0/24": 1.0, "192.30.253.0/24": 20290.0, "0.0.0.0/0": 1839.0} {"192.168.0.0/24": 14.0} {} {"0.0.0.0/0": 3398.0} {"0.0.0.0/0": 35.0} {}

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/iovisor/bcc/issues/1571#issuecomment-362875410, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDNm2zTr_ht124IKDlV53QiUxCrA1Bmks5tRRdPgaJpZM4R4COY .

[rmanyari]

Thanks for the feedback, I'll make changes accordingly and open a PR