Speed up validator key loading

[ajsutton]

Description

Investigate ways to speed up validator key loading. scrypt encrypted keystores are inherently slow to load but pbkdf2 should be faster and Teku is still pretty slow.

We should investigate what we can do to speed up loading pbkdf2 keys and what other options are available to load keys faster (e.g. alternative key storage formats etc).

[ohko4711]

echo "First: $(docker logs besu-teku-beacon-1 2>&1 | grep "Loaded validator key" | head -n1)" && echo "Last: $(docker logs  besu-teku-beacon-1  2>&1 | grep "Loaded validator key" | tail -n1)"
First: 2024-09-24 09:30:09.551 INFO  - Loaded validator key 10 of 4000.
Last: 2024-09-24 09:47:22.347 INFO  - Loaded validator key 4000 of 4000.

beacon-1     | 2024-09-24 09:47:17.188 INFO  - Loaded validator key 3980 of 4000.
beacon-1     | 2024-09-24 09:47:19.484 INFO  - Loaded validator key 3990 of 4000.
beacon-1     | 2024-09-24 09:47:22.347 INFO  - Loaded validator key 4000 of 4000.

almost 3s load 10 key, sir @ajsutton @rolfyone rolfyone Is there any plan to further optimize it? or take this part https://github.com/Consensys/teku/blob/fd599c4cfc587dcc9d35d2270760759e4901d61f/validator/client/src/main/java/tech/pegasys/teku/validator/client/loader/MultithreadedValidatorLoader.java#L83 as the configuration parameter

[rolfyone]

It's possible to re-encode the keys to be less strongly encrypted... I'd prefer to not load them into database without good encryption, i do think file storage is what we want...

Currently this has been on our backlog for a fair period of time and we've just had too many core changes to spend the time we need to dig into what else may be done to be honest. Your comment is the first since 2022...

[benjaminion]

If key loading time is an issue, you can try converting the keystores from Scrypt (the default format from most deposit tools) to PBKDF2 format. This tool comes from a reputable team. Teku will happily load these instead.

Scrypt is slow (and memory hungry) by design; PBKDF2 is much, much faster, but more easily brute-forced as a result. So you takes your choice.

[ohko4711]

If key loading time is an issue, you can try converting the keystores from Scrypt (the default format from most deposit tools) to PBKDF2 format. This tool comes from a reputable team. Teku will happily load these instead.

Scrypt is slow (and memory hungry) by design; PBKDF2 is much, much faster, but more easily brute-forced as a result. So you takes your choice.

@benjaminion thks for your advice! I recently ran a different client combination with the same amount of keys, and found that teku's performance has a magnitude gap compared with other cl clients.

[tbenr]

@ohko4711 are you referring Scrypt or PBKDF2 encrypted keys?

[ohko4711]

@ohko4711 are you referring Scrypt or PBKDF2 encrypted keys?

as user/validator, I just figured out that teku is slowly when loading a lot of keys than lighthouse/prysm, at that time I didn’t know lighthouse/prysm is Scrypt or PBKDF2

[benjaminion]

Aiui both Prysm and Lighthouse have an initial key-import process when you set them up, and then store the keys in their own format internally for future use. We took a decision not to do this for Teku since we didn't want to override the user's key-security model.

It could be revisited, but (1) I'd imagine that most set-ups with a lot of keys are likely using a remote signer such as Web3Signer, and (b) it can be worked around by converting the keystores to PBKDF2 for those whom it is a problem and are happy with the tradeoffs.