EPIC: Short term block propagation improvements

[evan-forbes]

There are currently a few changes that should be easy to incorporate while potentially have a large impact on the efficiency and effectiveness of block propagation. We should investigate these individually by running experiments against a control network. For each of the listed experiments, we need a report that measures the difference in:

• Time required to gossip the proposal block
• Total bandwidth used per MB of block data per peer per channel
• Percentage of block parts received more than once
• Percentage of bandwidth used by each reactor over time
• Percentage of heights that required more than one round to reach consensus

We need to know if these improvements along will enable the consensus layer to get to 32MB blocks without making more advanced changes in gossiping.

Experiments

• [ ] Further prioritize block part gossiping
• [ ] Use CAT
• [ ] reduced number of peers but increased bandwidth rate limits

Tasks

• [x] https://github.com/celestiaorg/celestia-app/issues/2033

[staheri14]

Since reactors may be responsible for different channel IDs, it would be more insightful to get "percentage of bandwidth used by each channel ID and per peer over time" (in replace of Percentage of bandwidth used by each reactor over time). Update: I see that this has been covered in an earlier item "Total bandwidth used per MB of block data per peer per channel", though, I am not sure how the "per MB of block data" impacts the calculation, mind elaborating?

One additional consideration: perhaps actual traffic per channel/reactor would be more informative than just percentages. Percentages show the bandwidth distribution among components, but these ratios can stay constant even if the actual bandwidth fluctuates. Monitoring the specific traffic values could provide a clearer picture of network health, unless the relationships between reactors/channel IDs are crucial for the analysis.

[staheri14]

I would also consider including the following metrics:

• Transaction throughput measured in bytes/second, calculated as block size/block time. This metric would allow us to compare the rate of traffic incurred by each P2P connection with the network's actual throughput. An analysis could then reveal the impact of varying block sizes on both the traffic overhead per P2P connection and the transaction throughput. It's noteworthy that sometimes traffic overhead may increase by growing block size while transaction throughput remains constant.

• The peak frequency of block part downloads and uploads. An examination similar to the analysis provided in link.