relab / hotstuff

MIT License
166 stars 52 forks source link

Use of gRPC for one way messages #80

Closed PasinduTennage closed 1 year ago

PasinduTennage commented 1 year ago

This repository uses gRPC as the messaging layer. gRPC by default uses the request-response pair architecture. However, protocols such as Hotstuff are written using Unicast abstraction.

In this implementation of Hotstuff, the gRPC request-response messages are used with google.protobuf.Empty as the return type. So to the best of my understanding, the response of each gRPC request is discarded.

While there is nothing wrong in this approach in the theoretical sense, there is a significant performance overhead; the message complexity of the implementation is 2 times the actual message complexity of Hotstuff (because of Empty response messages that are not required)

I kindly ask you to explain this design choice.

Thanks

meling commented 1 year ago

Hi there. You are right that we describe hotstuff messages as request-response messages for convenience and for the robustness provided by the protobuf and gRPC libraries.

However, what perhaps does not come across from a surface-level inspection of the code is that we use the relab/gorums library for most RPC calls. In particular, we use the multicast and unicast semantics for most messages, but also some quorum calls where that makes sense. The gorums library establishes connections with the replicas in a configuration, and sets up a gRPC stream onto which we dispatch the RPC messages (from hotstuff). In the case of unicast and multicast, these are actually “real” one-way messages, in that we don’t wait for the ACK; we use the gorums.WithNoSendWaiting() option to achieve this.

Let me know if this addresses your concern or you would like me to provide additional clarifications. Or if you have suggestions for improvements, e.g., to documentation that would address the concern. We have recently completed a paper (in submission) that includes some performance benchmarks; I'd be happy to share it with you if you reach out to me via email.

leandernikolaus commented 1 year ago

Here you can find a document describing the different options available in gorums: https://github.com/relab/gorums/blob/master/doc/method-options.md @meling This document should be linked from the user guide.

PasinduTennage commented 1 year ago

Hi @meling and @leandernikolaus

Thank you for the response, and it does clarify my question.

Regards