cncli leader log calculations using significant memory

[adavault]

Describe the bug cncli leader log calculations crashing cardano-node due to excessive memory use. Peak memory use for the node and cmcli is now 26.5GB from Epoch 282

To Reproduce Steps to reproduce the behavior:

1. Install cncli (via GuildOps tools in our case)
2. Pass the point in the epoch where leaderlogs calculations start
3. Wait for the leaderlogs calculations to conclude- memory use will peak as the results are written to database?

Expected behavior It should be possible to run cardano-node and cncli in less than 20GB

Screenshots If applicable, add screenshots to help explain your problem.

Server (please complete the following information): OS: ubuntu 20.04 server LTS Browser n/a Version 5.2.0

Additional context Physical memory on the servers is set to 20GB, by increasing swap it works correctly and does not crash the node. However passing beyond physical memory makes it possible that the node may be delayed if assigned leader slots coincide. Also opened bug report here: [https://github.com/cardano-community/cncli/issues/23]

[adavault]

Andrew has just got back and confirmed that this is an IOHK issue, not a cncli issue: https://github.com/input-output-hk/cardano-node/issues/3691

Suggest to add a note on prereqs or guild requirements that ~30GB RAM (can be physical and Swap) is now needed to run node and cncli reliably...

[Scitz0]

We will soon deploy a version that will utilize Koios API if you choose to do so for grabbing the needed info instead of querying node ledger for it that is very cpu and memory intensive.

[adavault]

Thanks for confirming, will look to move over to that

[rdlrt]

Will be fixed as part of #1572