Added comments to all examples, the main README and the Buffer to mention that Tower layering should be used to control the concurrency of the Consensus service if the implementation of request handling in call of the actual application service is asynchronous.
An example of an asynchronous implementation is here, which exhibited transaction reordering.
Implementation notes
I found that I needed to do the following combo to make it work with ABCI++ in v0.37:
Pass ServiceBuilder::new().buffer(block_max_msgs + 3).concurrency_limit(1).service(consensus) to the ServiceBuilder, where the buffer serves to allow all messages in a block lifecycle to be queued up without blocking Connection::run
Enforce the block_max_msgs setting in prepare_proposal and process_proposal so we don't get blocks which would violate that assumption.
Notably without .buffer, we got a deadlock because the responses weren't processed by Connection, and I suppose this applies to undersized buffers as well, which is why I added the proposal handling too.
This shouldn't be an issue with v0.38 which uses finalize_block.
I thought about adding it as an example to kvstore_037 but it would needlessly complicate it, as it's immune by the virtue of being synchronous.
Closes #44
Added comments to all examples, the main README and the
Bufferto mention that Tower layering should be used to control the concurrency of theConsensusservice if the implementation of request handling incallof the actual application service is asynchronous.An example of an asynchronous implementation is here, which exhibited transaction reordering.
Implementation notes
I found that I needed to do the following combo to make it work with ABCI++ in v0.37:
ServiceBuilder::new().buffer(block_max_msgs + 3).concurrency_limit(1).service(consensus)to theServiceBuilder, where thebufferserves to allow all messages in a block lifecycle to be queued up without blockingConnection::runblock_max_msgssetting inprepare_proposalandprocess_proposalso we don't get blocks which would violate that assumption..buffer, we got a deadlock because the responses weren't processed byConnection, and I suppose this applies to undersized buffers as well, which is why I added the proposal handling too.This shouldn't be an issue with v0.38 which uses
finalize_block.I thought about adding it as an example to
kvstore_037but it would needlessly complicate it, as it's immune by the virtue of being synchronous.