Benchmark the consensus reactor

[evan-forbes]

Currently we see that the consensus reactor takes significant amounts of times to process messages. During network tests, we can see a 50 node network with each node having 10-15 active peers can often result in a 10000 msg pileup in a buffer for envelopes.

To help fix this, we can create a benchmark for the receive portion of the consensus reactor. By creating specific scenarios, we should be able to identify and isolate more specific instances to increase the msg processing rate of the consensus reactor.

Acceptance Criteria

• [ ] Create a benchmark that simulates many different connected peers capable of submitting vast amounts of each consensus msg type.
• [ ] Either trace locally or otherwise keep trace of some simple stats. These stats should include msgs processed / s for each different channel and msg type.
• [ ] some basic benchmarks for large blocks.
• [ ] capability to simulate long ProcessProposal waits.

Bonus

• [ ] ability to easily toggle profiling

[evan-forbes]

Blocked on the results of the changes in https://github.com/cometbft/cometbft/pull/3211 make a large difference, if they do not and we still see large amounts of contention, then we continue with this issue. If not, then we move to unplanned.