High quantity of packets will cause gopacket to miss packet.

[lllby1993]

Table of Contents

1. problem
2. how to reproduce
3. other information

problem

I use gopacket to capture MySQL packets generated from a MySQL benchmark tool `mysqlslap`. I noticed that with a high quantity of MySQL packages, even from a single connection, will cause gopacket capture less packets than tcpdump does. Some packets are missed with no error reported.

how to reproduce

Use the following command to generate lots of MySQL queries:

mysqlslap --user=myuser --password=verysecretpass --host=solidcomputer --port=3306 --auto-generate-sql --concurrency=1 --number-of-queries 10000 --iterations=1

use gopacket and tcpdump to capture those queries and compare them.

When I use number-of-queries 10000, gopacket will print less packets than tcpdump. Some packets's TCP sequence number can be found in tcpdump's result but not in gopacket's result.

When I change number-of-queries to 500, gopacket will never miss any packet.

I tested with the default libpcap provided by Debian(version 1.8.1-3) and with the following information:

Example code

``` package main import ( "bufio" "flag" "io" "log" "net/http" "time" "github.com/google/gopacket" "github.com/google/gopacket/examples/util" "github.com/google/gopacket/layers" "github.com/google/gopacket/pcap" "github.com/google/gopacket/tcpassembly" "github.com/google/gopacket/tcpassembly/tcpreader" ) var iface = flag.String("i", "eth0", "Interface to get packets from") var fname = flag.String("r", "", "Filename to read from, overrides -i") var snaplen = flag.Int("s", 65535, "SnapLen for pcap packet capture") var filter = flag.String("f", "tcp and port 3306", "BPF filter for pcap") var logAllPackets = flag.Bool("v", false, "Logs every packet in great detail") // httpStreamFactory implements tcpassembly.StreamFactory type httpStreamFactory struct{} // httpStream will handle the actual decoding of http requests. type httpStream struct { net, transport gopacket.Flow r tcpreader.ReaderStream } func (h *httpStreamFactory) New(net, transport gopacket.Flow) tcpassembly.Stream { hstream := &httpStream{ net: net, transport: transport, r: tcpreader.NewReaderStream(), } go hstream.run() // Important... we must guarantee that data from the reader stream is read. // ReaderStream implements tcpassembly.Stream, so we can return a pointer to it. return &hstream.r } func (h *httpStream) run() { buf := bufio.NewReader(&h.r) for { _, err := http.ReadRequest(buf) if err == io.EOF { return } } } func main() { defer util.Run()() var handle *pcap.Handle var err error // Set up pcap packet capture if *fname != "" { log.Printf("Reading from pcap dump %q", *fname) handle, err = pcap.OpenOffline(*fname) } else { log.Printf("Starting capture on interface %q", *iface) handle, err = pcap.OpenLive(*iface, int32(*snaplen), true, pcap.BlockForever) } if err != nil { log.Fatal(err) } if err := handle.SetBPFFilter(*filter); err != nil { log.Fatal(err) } // Set up assembly streamFactory := &httpStreamFactory{} streamPool := tcpassembly.NewStreamPool(streamFactory) assembler := tcpassembly.NewAssembler(streamPool) log.Println("reading in packets") // Read in packets, pass to assembler. packetSource := gopacket.NewPacketSource(handle, handle.LinkType()) packets := packetSource.Packets() ticker := time.Tick(time.Minute) for { select { case packet := <-packets: // A nil packet indicates the end of a pcap file. if packet == nil { return } if *logAllPackets { log.Println(packet) } if packet.NetworkLayer() == nil || packet.TransportLayer() == nil || packet.TransportLayer().LayerType() != layers.LayerTypeTCP { log.Println("Unusable packet") continue } tcp := packet.TransportLayer().(*layers.TCP) log.Println(tcp.Seq) assembler.AssembleWithTimestamp(packet.NetworkLayer().NetworkFlow(), tcp, packet.Metadata().Timestamp) case <-ticker: // Every minute, flush connections that haven't seen activity in the past 2 minutes. assembler.FlushOlderThan(time.Now().Add(time.Minute * -2)) } } } ```

System Information

``` -- System Information: Debian Release: 9.6 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.9.0-6-amd64 (SMP w/2 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=zh_CN.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Init: systemd (via /run/systemd/system) ```

Go Information

``` go version go1.12.5 linux/amd64 GOARCH="amd64" GOBIN="" GOCACHE="~/.cache/go-build" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="~/go" GOPROXY="" GORACE="" GOROOT="/usr/local/go" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" GOROOT/bin/go version: go version go1.12.5 linux/amd64 GOROOT/bin/go tool compile -V: compile version go1.12.5 uname -sr: Linux 4.9.0-6-amd64 Distributor ID: Debian Description: Debian GNU/Linux 9.6 (stretch) Release: 9.6 Codename: stretch /lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Debian GLIBC 2.24-11+deb9u3) stable release version 2.24, by Roland McGrath et al. ```

other information

I tried with a minimal C code that captures packets with libpcap. It never miss any packet, which means that the problem is not related to libpcap.

When reading from a tcpdump's pcap file instead of capture from ethernet card, gopacket will never miss any packet whatever number-of-queries is set, which means that the problem is not related to the logic gopacket handles packets(such as tcpassembly).

I guess the problem is related to the "github.com/google/gopacket/pcap" module but I cannot locate it.

[notti]

I noticed that with a high quantity of MySQL packages, even from a single connection, will cause gopacket capture less packets than tcpdump does. Some packets are missed with no error reported. [...] When reading from a tcpdump's pcap file instead of capture from ethernet card, gopacket will never miss any packet whatever number-of-queries is set, which means that the problem is not related to the logic gopacket handles packets(such as tcpassembly).

Well this hints at packet processing being slower than the gap between two packets. Packet processing here is basically everything from reading the packet of the "wire" to tcp reassembly:

1. libpcap
2. gopacket/pcap
3. gopacket.PacketSource
4. gopacket/layers/* (whatever layers the packet contains)
5. gopacket/tcpassembly
6. gopacket/tcpassembly/tcpreader
7. bufio reader

In between a bit of code in your file. Any one of this, or just the sum could be too slow. But before I present some stuff first, how one should actually go about this: Use profiling.

Profiling

There is a really good introduction at https://blog.golang.org/profiling-go-programs how to profile go programs. Starting out with a CPU profile, looking at your main function, you could find out what takes how long and figure out what is too slow. One thing that uses lot's of CPU is memory allocation. So looking at a memory profile (with -alloc_space or -alloc_objects) can also help.

Now on to how gopacket can be sped up quiet a lot:

Speeding up gopacket

Do less work

One thing that speeds up everything in general is to simply not do something: Use a filter, so you only get the packets you need (e.g. in this case handle.SetBPFFilter("tcp port 3306"). This probably won't help much in your synthetic use case, since I imagine there wont' be much other traffic.

Avoid memory allocation

There is tons memory allocation going on:

• The packet source you use (gopacket.NewPacketSource(handle, handle.LinkType())) will call ReadPacketData on every packet you capture, which in turn allocates memory for the packet and copies the packet data into it (well depends a bit on the packet source - gopacket/pcap does this).
• The same packet source calls NewPacket on every packet. Since you didn't set any DecodeOptions, this NewPacket call will allocate another buffer and copy the whole packet data there.
• This NewPacket call will decode the whole packet and allocate layers for that - even those you didn't ask for (could be changed with the Lazy option)

So we basically allocated two buffers and copied the packet twice. Additionally, every layer gets allocated for every packet (in this case probably Ethernet, IPv4/IPv6, TCP).

First allocation + copy can be avoided with using ZeroCopyReadPacketData. Second one by setting NoCopy. BEWARE, now the buffer holding the packet data gets invalid when the next packet is read (it will hold the packet data of the next packet). You must be absolutely sure that you don't use the buffer in the future or interesting things will happen. tcpassembly looks ok from the source (in case the packet is out of order, it will copy the stuff it needs, and if it is in order, it will passt it along right away). BUT tcpreader will not play along with this nicely! This one will send the buffer to a channel, where it will be fetched by another goroutine, which might be after the buffer is overwritten by the next packet...

So either you use something different than tcpreader, or you have to allocate and copy something. But the payload would be enough - no need for the whole packet.

As for the layer allocation: Instead of NewPacket, you could use DecodingLayerParser (https://godoc.org/github.com/google/gopacket#hdr-Fast_Decoding_With_DecodingLayerParser). The layers you want are fixed anyway, so no need for having something that can handle everything. The given example in the url contains an example that fits your needs (tcp).

You could make it even faster, since you don't actually need the flexibility of the DecodingLayerParser and instead decode the packet manually (use DecodeFromBytes on ethernet layer with the packetdata, if no error, check if contained layer is ipv4, user DecodeFromBytes on ipv4 layer, ...).

Avoid printing stuff out for every packet

This is quite slow (log.Println(tcp.Seq)) so some other method for finding out if you caught everything would help (e.g., count packets and write out a summary at the end).

Use profiling

See above. Best way to know what is actually slow is to profile the program. You might be optimizing the wrong part of your code. Optimized code will always be more complicated and harder to read - so definitely start out with a simple version, if it is too slow, profile, and then make the parts that are too slow faster.

So I would start out with the ZeroCopy-stuff, DecodingLayerParser or manual decoding, and not use tcpreader. If you have to use tcpreader, then you at least have to do one copy (e.g., normal ReadPacketData, which would do the copy for you and DecodingLayerParser).