relab / gorums

Gorums simplify fault-tolerant quorum-based protocols
MIT License
138 stars 14 forks source link

Allow server handlers to use the return statement to send back replies #146

Closed johningve closed 3 years ago

johningve commented 3 years ago

This brings back elements of the older server API where server handlers are simple Go functions that take the request object as a parameter and return the response object (like in gRPC).

In our current API, each handler receives a function to use for sending back the response. The reason for this API is to allow the server handlers to run synchronously on the server, while at the same time making it possible to start new goroutines from the handlers. This made it possible for some of the request handling code to call the "response" function at a later time to send back the reply. The example below shows one such handler:

func RPCCall(ctx context.Context, req *Request, out func(*Response, error) {
    // perform operations that must be in-order
    ...
    go func() {
        // perform operations that can be concurrent
        ...
        // send the response
        out(response, nil)
    }()
}

With this change, the handlers instead run in separate goroutines, but with a mutex lock on the server. The handler must release the shared mutex lock before another request can be handled. The handler can either release the lock by returning, or by calling a "release" function on the server context. The example below shows the new server API:

func RPCCall(ctx gorums.ServerCtx, req *Request) (*Response, error) {
    // perform operations that must be in-order
    ...
    // release the shared lock, which allows another handler to be started.
    ctx.Release()
    // perform operations that can be concurrent
    ...
    // return the response
    return response, nil
}

The "ctx" object is a special server context that wraps a normal context.Context. It is used by the handler to release the shared mutex lock of the server. The handler may call this manually, as shown in the example above, but it will also be executed automatically once the handler has returned and the response has been sent.

The implementation looks like this (from server.go):


    // Start with a locked mutex
    s.mut.Lock()
    defer s.mut.Unlock()

    for {
        req := newMessage(requestType)
        err := srv.RecvMsg(req)
        if err != nil {
            return err
        }
        if handler, ok := s.handlers[req.Metadata.Method]; ok {
            // We start the handler in a new goroutine in order to allow multiple handlers to run concurrently.
            // However, to preserve request ordering, the handler must unlock the shared mutex when it has either
            // finished, or when it is safe to start processing the next request.
            go handler(ServerCtx{Context: ctx, once: new(sync.Once), mut: &s.mut}, req, finished)
            // Wait until the handler releases the mutex.
            s.mut.Lock()
        }
    }

@meling This PR modified a lot of files, but the most important are server.go and template_server.go. Also, there are two commits to this PR. The first adds a release() function to the server handlers, and then the second commit moves this release function into a ServerCtx object. I quite like the ServerCtx approach because it "hides" the response function handlers where it is not needed.