Implement Resource Pooling for Block Range Merkle maps

[jpraynaud]

Why

In order to improve the throughput of transactions proof generation on the aggregator REST API, we want to implement a resource pool for the block range Merkle maps which is the proof operation that takes most of the time so far. During the performance assessment phase of #1687, we have achieved x500 acceleration factor with this optimization. Also, this will allow throttling of the resources allocated to the proof generation on the aggregator so that the certificate production is not penalized.

What

Implemement a resource pool for block range Merkle maps (pool size is a configuration parameter of the aggregator).

How

• [x] Implement a resource pool for block range Merkle map:
  • [x] Resources automatically returned to the pool
  • [x] Stale resource detection on resource returned to the pool
  • [x] Non blocking refresh of the resources in the pool
• [x] Warm up the resource pool at node startup
• [x] Refresh the resource pool of block range Merkle maps when a new Cardano transaction artifact is produced
• [x] Use the resource pool as cache for block range Merkle map in the prover
• [x] Make the size of the prover cache pool configurable
• [x] Implement the configuration of the prover cache pool size in the infrastructure
• [x] Implement the configuration of the prover cache pool size in the CI/CD
• [x] Update GitHub environments with new configuration
• [x] Document the impact in terms of memory consumption for the aggregator
• [x] Document a benchmark with performances improvements

[jpraynaud]

Benchmark results

Total Requests	Concurrency	Transactions / Request	Pool (x1): Request/s	Pool (x5): Request/s	Pool (x10): Request/s	Pool (x25): Request/s	Pool (x50): Request/s
1000	100	1	39	156	213	229	237
1000	100	2	28	110	152	169	171
1000	100	3	21	83	116	129	128
1000	100	4	18	70	96	104	105
1000	100	5	15	59	82	87	86
1000	100	6	0	0	0	0	0
1000	100	7	0	0	0	0	0
1000	100	8	0	0	0	0	0
1000	100	9	0	0	0	0	0
1000	100	10	0	0	0	0	0

Running on Linux / 8 cores / 64 GB RAM / 2 TB SSD

Memory impact

~240MB / new pool resource

Pooled Resources	Memory / GB
1	0
5	1
10	3
25	6
50	13
100	25

Conclusions

The performances are considerably improved with the implementation of resource pooling for Block Ranges Merkle maps. We recommend to use a size of the pool with a value between 25 and 50 which gives good trade-off between memory usage and performances gain. This value is subject to change as we suspect that other factors are limiting the throughput which drastically drops with more than 5 transactions to certifiy per request.