search

Consensys / web3signer

Web3Signer is an open-source signing service capable of signing on multiple platforms (Ethereum1 and 2, Filecoin) using private keys stored in an external vault, or encrypted on a disk.
https://docs.web3signer.consensys.net/
Apache License 2.0
201 stars 78 forks source link

Time to load keys #914

Closed yorickdowne closed 1 year ago

yorickdowne commented 1 year ago

web3signer w/ 10k keys that were created by staking-deposit-cli, Holesky, loaded via keymanager API

I am wondering whether the time to load keys can be improved on startup. Every startup web3signer takes ~20 mins

eth-holesky-web3signer-1  | 2023-09-15 15:42:24.472+00:00 | ForkJoinPool.commonPool-worker-2 | INFO  | SignerLoader | 9980 signing metadata processed
eth-holesky-web3signer-1  | 2023-09-15 15:42:26.832+00:00 | pool-2-thread-1 | INFO  | SignerLoader | 9990 signing metadata processed
eth-holesky-web3signer-1  | 2023-09-15 15:42:29.404+00:00 | ForkJoinPool.commonPool-worker-10 | INFO  | SignerLoader | 10000 signing metadata processed
eth-holesky-web3signer-1  | 2023-09-15 15:42:29.979+00:00 | pool-2-thread-1 | INFO  | SignerLoader | Total Artifact Signer loaded via configuration files: 10000
eth-holesky-web3signer-1  | Error count 0
eth-holesky-web3signer-1  | Time Taken: 00:19:14.756.
jframe commented 1 year ago

Does it take the same time to load keys using the keyManager API as does loading by restarting web3signer?

The keys that are generated using the staking-deposit-cli can take a long time to load as encrypted using scrypt. There might not be much we do and can still take a look.

yorickdowne commented 1 year ago

I believe it does take just as long, yes.

I know scrypt decode takes a minute. Clients like Teku store this in an optimized format somehow while still keeping it encrypted - maybe you can crib from how it's handled there?

usmansaleem commented 1 year ago

@yorickdowne the main reason for slow loading at start up is due to scrypt encryption on the v4 wallet files. Teku also doesn't convert any of the v4 imported parameters. We also identify one potential enhancement during the import phase though and will be incorporating it via #921. You should be able to convert scrypt to pbkdf2 to decrease the decryption time, but with the caveat of lower security. Let us know if you need more information around how to convert or lower the v4 file parameters, it would need a small Java program to achieve it.

yorickdowne commented 1 year ago

Got it. Yes that Java program for conversion would be helpful.

usmansaleem commented 1 year ago

@yorickdowne I created a small Java program that should help you to convert existing SCRYPT v4 keystore files to either SCRYPT or PBKDF2 with lower n or c respectively. https://github.com/usmansaleem/v4keystore_converter . Let me know if you need more information.

yorickdowne commented 1 year ago

Thank you, amazing! Will give this a try next week.

usmansaleem commented 1 year ago

Closing the ticket. Feel free reopen the ticket or reach out on Consensys Discord channel if you have further queries regarding the java program. Discord handle: usmans.info.

yorickdowne commented 1 year ago

Great success! Converting to pbkdf2 with c=10 loads keys almost instantly. Thank you!