graceful shutdown

[yosiat]

Hi,

I wrote a redcon server and I need to support graceful shutdown, from what I observed calling "Close" only closes the server connection and don't wait for in-memory requests to flush.

Is there a recommended way to implement graceful shutdown?

[tidwall]

Hi, you're right. Calling Close only closes the network listener, which stop the server from accepting new connections.

But existing connection will continue to run until they're network socket has been closed.

The only way I can think of with the current implementation is to track the connections using the accept and close callbacks and to wait for the all connections to be closed using a WaitGroup. Also to use the SetIdleClose function to automatically close idle connections.

For example, here we'll create a new server, which will automatically close after 5 seconds.

func main() {
    // Create a new Server
    var wg sync.WaitGroup // connection wait group
    var closed int32      // atomic flag
    s := redcon.NewServer(":6380",
        // handler
        func(conn redcon.Conn, cmd redcon.Command) {
            if atomic.LoadInt32(&closed) != 0 {
                // server closed, close connection
                conn.Close()
            } else {
                // TODO: handle incoming command
                conn.WriteString("hello")
            }
        },
        // accept
        func(conn redcon.Conn) bool {
            if atomic.LoadInt32(&closed) != 0 {
                // Server closed, do not accept this connection
                return false
            }
            // Add connection to a wait group
            wg.Add(1)
            return true
        },
        // close
        func(conn redcon.Conn, err error) {
            // Remove connection from wait group
            wg.Done()
        },
    )
        // Set a max amount of time a connection can stay idle.
    s.SetIdleClose(time.Second * 10)
    go func() {
        // Close the server after an minute
        for i := 5; i > 0; i-- {
            println("closing in", i)
            time.Sleep(time.Second)

        }
        // Set the closed flag and wait for the connections to be closed.
        atomic.StoreInt32(&closed, 1)
        println("waiting for connections to close")
        wg.Wait()
        // No more live connections, close the server
        s.Close()
    }()
    if err := s.ListenAndServe(); err != nil {
        log.Fatal(err)
    }
    // The server is now closed
    println("closed")
}

[tidwall]

Now if you connect to the server using redis-cli within the 5 seconds. You will see

closing in 5
closing in 4
closing in 3
closing in 2
closing in 1
waiting for connections to close

a pause up to 10 second because of the SetIdleClose, then:

closed

[tidwall]

Ideally this logic would exist in the library. Maybe this is something that would work with #52

[yosiat]

@tidwall tested this locally and it works, thanks for the explanation and proposed solution!

Would be interested to see how graceful solution can be done with context, will follow to see that :)