Closed moshe-blox closed 3 months ago
After some consideration, I would rather avoid using a flag that is as focused as this. As the size of the beacon chain grows it's likely that we will have additional situations where we will make time/space trade-offs, so a more general flag would make sense.
So could we change the flag to be something like "WithReducedMemoryUsage(bool)", with the parameter defaulting to false
(and a note that this may result in longer response times as a result)? That gives us a level of flexibility in future without needing additional flags, or to deprecate flags that relate to implementation details as/when the beacon API evolves.
In this particular situation, if the request is made for reduced memory usage then the chunked method will be used at for all requests other than a request for all validators, and otherwise the existing logic will be used.
@moshe-blox are you okay to alter the PR as per the above comment?
@mcdee I have opened another PR that uses the approach you described: https://github.com/attestantio/go-eth2-client/pull/121
Closing in favor of #121
Continuing from https://github.com/attestantio/go-eth2-client/pull/100, we've found that
BeaconState
calls in mainnet allocate ~1170 MiB memory and require ~520 MiB memory from the OS to execute.Our mainnet SSV nodes on Kubernetes now request up to 2 GB memory (in spikes), when they previously only needed up to 0.7 GB, yet they're still querying the same number of validators (~3000).
I'm not ruling out potential optimizations which can be done in
validatorsFromState
, yet I believe allowing go-eth2-client users to opt-out ofBeaconState
calls for slower operation in return for less memory usage is a reasonable trade-off nonetheless.Reproducible:
Output on a Linux machine with a mainnet Beacon node: