Ensure network stats also work for IPv6

[brharrington]

Some like net.tcp.memory only seem to report usage for IPv4. As more important workloads move to IPv6 this gap can be quite misleading as it likely isn't apparent to the user.

[copperlight]

It looks like there is a /proc/net/sockstat6 file, which is the IPv6 equivalent of /proc/net/sockstat, which supplies the data for net.tcp.memory. However, on a sample system, the data it contains is different and does not appear to report memory use.

$ cat /proc/net/sockstat
sockets: used 343
TCP: inuse 18 orphan 0 tw 29 alloc 71 mem 3
UDP: inuse 4 mem 2
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

$ cat /proc/net/sockstat6
TCP6: inuse 33
UDP6: inuse 3
UDPLITE6: inuse 0
RAW6: inuse 1
FRAG6: inuse 0 memory 0

The Linux Documentation Project entry for /proc/net/sockstat6 is not complete:

https://tldp.org/HOWTO/Linux+IPv6-HOWTO/ch11s04.html

The kernel docs are thin on this topic as well:

https://github.com/torvalds/linux/blob/0bb80ecc33a8fb5a682236443c1e740d5c917d1d/Documentation/filesystems/proc.rst#L1276

The kernel source for /proc/net/sockstat6 shows that only inuse sockets are reported:

https://github.com/torvalds/linux/blob/0bb80ecc33a8fb5a682236443c1e740d5c917d1d/net/ipv6/proc.c#L34

Contrast this with the kernel source for /proc/net/sockstat, which shows that mem is reported:

https://github.com/torvalds/linux/blob/0bb80ecc33a8fb5a682236443c1e740d5c917d1d/net/ipv4/proc.c#L51

Some discussion here indicates that TCP4 and TCP6 are misnomers - they really mean TCP/IPv4 and TCP/IPv6 - the Layer 4 protocol is the same. I guess that this means that /proc/net/sockstat remains the official kernel record of TCP memory used?

https://unix.stackexchange.com/questions/329115/ipv6-over-tcp-or-tcp6

TCP4 or TCP6 protocols don't exist. They can be used as a shorthand to indicate respectively TCP with IPv4 and TCP with IPv6, but that's an abuse of language -- the protocol used is always TCP.

Due to the separation of layers in the ISO/OSI model, the TCP segment (level 4) is always the same whether it's accompanying a IPv4 or IPv6 packet (level 3).

The only thing that changes in the TCP segment is the Checksum field

Some commentary here confirms that /proc/net/sockstat is the place to look for TCP memory usage, although it does not discuss IPv4 versus IPv6.

https://unix.stackexchange.com/questions/419518/how-to-tell-how-much-memory-tcp-buffers-are-actually-using

[copperlight]

As a verification check, a small program to compare the data in /proc/net/sockstat and the output from ss:

#!/bin/python3

import re
import subprocess

def sockstat():
    with open('/proc/net/sockstat') as f:
        data = f.read()
        lines = data.splitlines()
        for line in lines:
            if line.startswith("TCP"):
                parts = line.split(' ')
                return int(parts[-1]) * 4096

def stripalpha(s):
    return re.sub(r'^[a-z]+', '', s)

def ss(version):
    response = subprocess.run(
            ['ss', '-t', '-m', '--ipv{}'.format(version)],
            capture_output = True,
            text = True)
    lines = response.stdout.splitlines()

    stats = {
        'count': 0,
        'rmem_alloc': 0,
        'wmem_alloc': 0,
        'fwd_alloc': 0,
        'wmem_queued': 0,
        'opt_mem': 0,
        'backlog': 0
    }
    for line in lines:
        if 'skmem' in line:
            parts = line.strip()[7:-1].split(',')
            stats['count'] += 1
            stats['rmem_alloc'] += int(stripalpha(parts[0]))
            stats['wmem_alloc'] += int(stripalpha(parts[2]))
            stats['fwd_alloc'] += int(stripalpha(parts[4]))
            stats['wmem_queued'] += int(stripalpha(parts[5]))
            stats['opt_mem'] += int(stripalpha(parts[6]))
            stats['backlog'] += int(stripalpha(parts[7]))

    total = sum(stats.values()) - stats['count']
    print('')
    print('ss - TCP/IPv{}'.format(version))
    print('- count:       {:,}'.format(stats['count']))
    print('- rmem_alloc:  {:,}'.format(stats['rmem_alloc']))
    print('- wmem_alloc:  {:,}'.format(stats['wmem_alloc']))
    print('- fwd_alloc:   {:,}'.format(stats['fwd_alloc']))
    print('- wmem_queued: {:,}'.format(stats['wmem_queued']))
    print('- opt_mem:     {:,}'.format(stats['opt_mem']))
    print('- backlog:     {:,}'.format(stats['backlog']))
    print('- total:       {:,}'.format(total))

print('sockstat: {:,}'.format(sockstat()))
ss(4)
ss(6)

There is a difference, but the values are in the same ballpark.

# ./tcp_memory.py
sockstat: 58,155,008

ss - TCP/IPv4
- count:       2
- rmem_alloc:  0
- wmem_alloc:  0
- fwd_alloc:   0
- wmem_queued: 0
- opt_mem:     0
- backlog:     0
- total:       0

ss - TCP/IPv6
- count:       1,446
- rmem_alloc:  1,280
- wmem_alloc:  0
- fwd_alloc:   64,335
- wmem_queued: 61,251,505
- opt_mem:     0
- backlog:     0
- total:       61,317,120