[Question]: When receiving UDP broadcast packets, the same packet may be received multiple times

[czy0538]

Update 2023-02-27 14:07： I found that this issue is related to the gnet.WithMulticore(multicore) option .https://github.com/panjf2000/gnet/issues/442#issuecomment-1445759063

Questions with details

Short description of the issue:

When sending broadcast packets to a specific port and listening on that port, the gnet server receives multiple duplicate packets, while Wireshark capture, the net package, and Python program only receive one packet.

当发送 UDP 广播包时，gnet server 会重复的收到同一个包，net 、python 不会出现此问题，wireshark 也只会捕获一个包。

Some potentially useful information:

• macOS 13.2.1 arm64
• go1.20.1
• gnet v 2.2.5

Log

gnet：

=== RUN   TestInitUDPServer
{"time":"2023-02-26T21:07:41.722233+08:00","level":"INFO","msg":"init udp server","port":9001,"multicore":true}
{"time":"2023-02-26T21:07:41.72259+08:00","level":"INFO","msg":"UDP server engine is ready for accepting connections","addr":"udp://:9001","multicore":true}
{"time":"2023-02-26T21:07:54.285532+08:00","level":"DEBUG","msg":"udp server:","received":"Hello world"}
{"time":"2023-02-26T21:07:54.285574+08:00","level":"DEBUG","msg":"udp server:","received":"Hello world"}
{"time":"2023-02-26T21:07:54.28558+08:00","level":"DEBUG","msg":"udp server:","received":"Hello world"}
{"time":"2023-02-26T21:07:54.285584+08:00","level":"DEBUG","msg":"udp server:","received":"Hello world"}
{"time":"2023-02-26T21:07:54.285589+08:00","level":"DEBUG","msg":"udp server:","received":"Hello world"}
{"time":"2023-02-26T21:07:54.285595+08:00","level":"DEBUG","msg":"udp server:","received":"Hello world"}
{"time":"2023-02-26T21:07:54.285629+08:00","level":"DEBUG","msg":"udp server:","received":"Hello world"}
{"time":"2023-02-26T21:07:54.285653+08:00","level":"DEBUG","msg":"udp server:","received":"Hello world"}

wireshark

net

❯ go run "/Users/czy0538/Temp/testgo/main.go"
172.20.143.142:55502 sent this: Hello world

Code snippets (optional)

gnet server

func (es *Server) OnTraffic(c gnet.Conn) gnet.Action {
    buf, _ := c.Next(-1)
    newBuf := make([]byte, len(buf))
    copy(newBuf, buf)

    go func(c gnet.Conn, newBuf []byte) {
        ms := message.Message{
            ID:   "172.20.10.2",
            Data: newBuf,
        }
        es.ch <- ms
        slog.Debug("udp server:", "received", string(newBuf))
    }(c, newBuf)
    return gnet.None
}

func initUdpServer(port int, multicore bool, ch message.MessageChan, ifaceName string) error {
    slog.Info("init udp server", "port", port, "multicore", multicore)
    udpServer := &Server{
        addr:      fmt.Sprintf("udp://:%d", port),
        multicore: multicore,
        ch:        ch,
    }
    iface, _ := net.InterfaceByName(ifaceName)
    err := gnet.Run(udpServer, udpServer.addr, gnet.WithMulticore(multicore), gnet.WithMulticastInterfaceIndex(iface.Index))
    if err != nil {
        slog.Error("init UDP server error", err)
        return err
    }
    return nil
}
func InitUdpServer(ch message.MessageChan) error {
    return initUdpServer(9001, true, ch, "en0")
}

Code for sending UDP broadcast packets

func SendUDPBroadcast( port int, data []byte) error {
    msg := make([]byte, len(data))
    copy(msg, data)
    broadcastAddr := &net.UDPAddr{
        IP:   net.IPv4bcast,
        Port: port,
    }
    conn, err := net.DialUDP("udp4", nil, broadcastAddr)
    if err != nil {
        slog.Error("creat UDP client error", err)
        return err
    }
    defer conn.Close()
    _, err = conn.Write(msg)
    if err != nil {
        slog.Error("send UDP broadcast message error", err)
        return err
    }
    slog.Debug("send UDP broadcast message success", "local addr", conn.LocalAddr().String(), "remote addr", conn.RemoteAddr().String())
    return nil
}

net code

func main() {
    pc, err := net.ListenPacket("udp4", ":9001")
    if err != nil {
        panic(err)
    }
    defer pc.Close()

    buf := make([]byte, 1024)
    for {
        n, addr, err := pc.ReadFrom(buf)
        if err != nil {
            panic(err)
        }

        fmt.Printf("%s sent this: %s\n", addr, buf[:n])
    }

}

[czy0538]

I found that this issue is related to the gnet.WithMulticore(multicore) option. When this option is set to true, this issue occurs, and the number of duplicated messages is equal to the number of CPU cores I have, but it is not affected by GOMAXPROCS.

I tested it using the simplest echo program, and here are the test results：

❯ GOMAXPROCS=1 go run ./echo.go --multicore=false
1
Hello world
^Csignal: interrupt
❯ GOMAXPROCS=1 go run ./echo.go --multicore=true
1
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
^Csignal: interrupt
❯ go run ./echo.go --multicore=true
8
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
^Csignal: interrupt

Here are my test code:

package main

import (
    "flag"
    "fmt"
    "log"
    "runtime"

    "github.com/panjf2000/gnet/v2"
)

type echoServer struct {
    gnet.BuiltinEventEngine
}

func (es *echoServer) OnTraffic(c gnet.Conn) gnet.Action {
    data, _ := c.Next(-1)
    newData := make([]byte, len(data))
    copy(newData, data)
    go func(data []byte) {
        fmt.Println(string(data))
    }(newData)
    // c.Write(data)
    return gnet.None
}

func main() {
    fmt.Println(runtime.GOMAXPROCS(runtime.NumCPU() - 1))

    var port int
    var multicore, reuseport bool

    // Example command: go run echo.go --port 9000 --multicore=true --reuseport=true
    flag.IntVar(&port, "port", 9000, "--port 9000")
    flag.BoolVar(&multicore, "multicore", false, "--multicore true")
    flag.BoolVar(&reuseport, "reuseport", false, "--reuseport true")
    flag.Parse()
    echo := new(echoServer)
    log.Fatal(gnet.Run(echo, fmt.Sprintf("udp://:%d", port), gnet.WithMulticore(multicore), gnet.WithReusePort(reuseport)))
}

[panjf2000]

Then this might be caused by Thundering Herd, now that we have a clue about the issue, I'll try to dive deep and submit a fix, thanks for catching this.

[czy0538]

Thanks for your hard and brilliant work. As a beginner in Go, I have learned a lot from your articles and code.

[panjf2000]

Would this issue reproduce on Linux? @czy0538

[czy0538]

@panjf2000 My personal Linux server doesn't have this issue, but it only has one core, which may affect the test.

[panjf2000]

I found that this issue is related to the gnet.WithMulticore(multicore) option. When this option is set to true, this issue occurs, and the number of duplicated messages is equal to the number of CPU cores I have, but it is not affected by GOMAXPROCS.

I tested it using the simplest echo program, and here are the test results：

❯ GOMAXPROCS=1 go run ./echo.go --multicore=false
1
Hello world
^Csignal: interrupt
❯ GOMAXPROCS=1 go run ./echo.go --multicore=true
1
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
^Csignal: interrupt
❯ go run ./echo.go --multicore=true
8
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
^Csignal: interrupt

Here are my test code:

package main

import (
  "flag"
  "fmt"
  "log"
  "runtime"

  "github.com/panjf2000/gnet/v2"
)

type echoServer struct {
  gnet.BuiltinEventEngine
}

func (es *echoServer) OnTraffic(c gnet.Conn) gnet.Action {
  data, _ := c.Next(-1)
  newData := make([]byte, len(data))
  copy(newData, data)
  go func(data []byte) {
      fmt.Println(string(data))
  }(newData)
  // c.Write(data)
  return gnet.None
}

func main() {
  fmt.Println(runtime.GOMAXPROCS(runtime.NumCPU() - 1))

  var port int
  var multicore, reuseport bool

  // Example command: go run echo.go --port 9000 --multicore=true --reuseport=true
  flag.IntVar(&port, "port", 9000, "--port 9000")
  flag.BoolVar(&multicore, "multicore", false, "--multicore true")
  flag.BoolVar(&reuseport, "reuseport", false, "--reuseport true")
  flag.Parse()
  echo := new(echoServer)
  log.Fatal(gnet.Run(echo, fmt.Sprintf("udp://:%d", port), gnet.WithMulticore(multicore), gnet.WithReusePort(reuseport)))
}

Please also share the code of client, I'm trying to reproduce this issue on my multi-cores linux server, thanks! @czy0538

[czy0538]

I found that this issue is related to the gnet.WithMulticore(multicore) option. When this option is set to true, this issue occurs, and the number of duplicated messages is equal to the number of CPU cores I have, but it is not affected by GOMAXPROCS. I tested it using the simplest echo program, and here are the test results：

❯ GOMAXPROCS=1 go run ./echo.go --multicore=false
1
Hello world
^Csignal: interrupt
❯ GOMAXPROCS=1 go run ./echo.go --multicore=true
1
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
^Csignal: interrupt
❯ go run ./echo.go --multicore=true
8
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
Hello world
^Csignal: interrupt

Here are my test code:

package main

import (
    "flag"
    "fmt"
    "log"
    "runtime"

    "github.com/panjf2000/gnet/v2"
)

type echoServer struct {
    gnet.BuiltinEventEngine
}

func (es *echoServer) OnTraffic(c gnet.Conn) gnet.Action {
    data, _ := c.Next(-1)
    newData := make([]byte, len(data))
    copy(newData, data)
    go func(data []byte) {
        fmt.Println(string(data))
    }(newData)
    // c.Write(data)
    return gnet.None
}

func main() {
    fmt.Println(runtime.GOMAXPROCS(runtime.NumCPU() - 1))

    var port int
    var multicore, reuseport bool

    // Example command: go run echo.go --port 9000 --multicore=true --reuseport=true
    flag.IntVar(&port, "port", 9000, "--port 9000")
    flag.BoolVar(&multicore, "multicore", false, "--multicore true")
    flag.BoolVar(&reuseport, "reuseport", false, "--reuseport true")
    flag.Parse()
    echo := new(echoServer)
    log.Fatal(gnet.Run(echo, fmt.Sprintf("udp://:%d", port), gnet.WithMulticore(multicore), gnet.WithReusePort(reuseport)))
}

Please also share the code of client, I'm trying to reproduce this issue on my multi-cores linux server, thanks! @czy0538

I tried both a Python script and a simple Go program, and here is the code.

import socket

# 目标IP地址和端口
UDP_IP = "255.255.255.255"
UDP_PORT = 9000
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

MESSAGE = b"Hello world"
sock.sendto(MESSAGE, (UDP_IP, UDP_PORT))

sock.close()

@panjf2000

[panjf2000]

I think this issue is not caused by thunder herd but by SO_BROADCAST, gnet starts multiple listeners with reuseport flag when multicore=true or the protocol is UDP and UDP broadcast will send the frame to all listeners, which results in multiple received messages. The reason why the std net didn't encounter this issue is because you only start one listener for it, you'll observe the same phenomenon in this issue if you start more than one UDP listener with reuseport flag.

[czy0538]

I think this issue is not caused by thunder herd but by SO_BROADCAST, gnet starts multiple listeners with reuseport flag when multicore=true or the protocol is UDP and UDP broadcast will send the frame to all listeners, which results in multiple received messages. The reason why the std net didn't encounter this issue is because you only start one listener for it, you'll observe the same phenomenon in this issue if you start more than one UDP listener with reuseport flag.

Ah, I see! Thanks for your answer. It seems that I need to set multicore = false when facing broadcasting, or add some filtering at the application layer. Thanks again for your explanation.