Slow key loading in Local Disk

[YuXiaoCoder]

My node has 8 CPUs, I added JAVA_OPTS, there are 298 Keys in total, it takes 1 minute to finish loading, I refer to this ISSUE (https://github.com/Consensys/web3signer/issues/786) and found that others load 5000 Keys only It takes 7 seconds, I'm not sure what I'm configuring wrong.

/opt/eth-validatormain/core/web3signer/bin/web3signer --config-file=/mnt/eth-validatormain/conf/web3signer.yaml eth2

2023-06-29 21:03:52.658+08:00 | main | INFO  | Web3SignerApp | Web3Signer has started with args --config-file=/mnt/eth-validatormain/conf/web3signer.yaml,eth2
2023-06-29 21:03:52.692+08:00 | main | INFO  | Web3SignerApp | Version = web3signer/v23.6.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-11
2023-06-29 21:03:53.719+08:00 | main | INFO  | Eth2SubCommand | Network: mainnet
Spec Name: PHASE0, Fork Epoch: 0, First Slot: 0
Spec Name: ALTAIR, Fork Epoch: 74240, First Slot: 2375680
Spec Name: BELLATRIX, Fork Epoch: 144896, First Slot: 4636672
Spec Name: CAPELLA, Fork Epoch: 194048, First Slot: 6209536

2023-06-29 21:03:54.132+08:00 | pool-2-thread-1 | INFO  | SignerLoader | Loading signer configuration metadata files from /mnt/eth-validatormain/node/web3signer/keys
2023-06-29 21:03:54.954+08:00 | pool-2-thread-1 | INFO  | SignerLoader | Signer configuration metadata files read in memory 298 in 00:00:00.817
2023-06-29 21:03:55.140+08:00 | pool-2-thread-1 | INFO  | SignerLoader | Converting signing metadata to Artifact Signer using parallel streams ...
2023-06-29 21:03:56.777+08:00 | ForkJoinPool-1-worker-9 | INFO  | teku-status-log | Using optimized BLST library
2023-06-29 21:03:56.787+08:00 | ForkJoinPool-1-worker-9 | INFO  | BLS | BLS: loaded BLST library
2023-06-29 21:03:56.793+08:00 | ForkJoinPool-1-worker-9 | INFO  | SignerLoader | 10 signing metadata processed
2023-06-29 21:03:59.003+08:00 | ForkJoinPool-1-worker-7 | INFO  | SignerLoader | 20 signing metadata processed
2023-06-29 21:04:01.268+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 30 signing metadata processed
2023-06-29 21:04:03.593+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 40 signing metadata processed
2023-06-29 21:04:05.845+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 50 signing metadata processed
2023-06-29 21:04:07.970+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 60 signing metadata processed
2023-06-29 21:04:10.057+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 70 signing metadata processed
2023-06-29 21:04:12.309+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 80 signing metadata processed
2023-06-29 21:04:14.542+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 90 signing metadata processed
2023-06-29 21:04:16.895+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 100 signing metadata processed
2023-06-29 21:04:19.138+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 110 signing metadata processed
2023-06-29 21:04:21.205+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 120 signing metadata processed
2023-06-29 21:04:23.211+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 130 signing metadata processed
2023-06-29 21:04:25.246+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 140 signing metadata processed
2023-06-29 21:04:27.445+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 150 signing metadata processed
2023-06-29 21:04:29.779+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 160 signing metadata processed
2023-06-29 21:04:31.855+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 170 signing metadata processed
2023-06-29 21:04:33.808+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 180 signing metadata processed
2023-06-29 21:04:35.939+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 190 signing metadata processed
2023-06-29 21:04:38.127+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 200 signing metadata processed
2023-06-29 21:04:40.168+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 210 signing metadata processed
2023-06-29 21:04:42.342+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 220 signing metadata processed
2023-06-29 21:04:44.515+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 230 signing metadata processed
2023-06-29 21:04:46.507+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 240 signing metadata processed
2023-06-29 21:04:48.527+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 250 signing metadata processed
2023-06-29 21:04:50.838+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 260 signing metadata processed
2023-06-29 21:04:53.062+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 270 signing metadata processed
2023-06-29 21:04:55.297+08:00 | ForkJoinPool-1-worker-3 | INFO  | SignerLoader | 280 signing metadata processed
2023-06-29 21:04:59.027+08:00 | ForkJoinPool-1-worker-9 | INFO  | SignerLoader | 290 signing metadata processed
2023-06-29 21:05:03.738+08:00 | pool-2-thread-1 | INFO  | SignerLoader | Total Artifact Signer loaded via configuration files: 298
Error count 0
Time Taken: 00:01:08.782.
2023-06-29 21:05:03.741+08:00 | pool-2-thread-1 | INFO  | Eth2Runner | Bulk loading keys from local keystores ...
2023-06-29 21:05:03.747+08:00 | pool-2-thread-1 | INFO  | Eth2Runner | Keys loaded from local keystores: [0], with error count: [0]
2023-06-29 21:05:03.752+08:00 | pool-2-thread-1 | INFO  | DefaultArtifactSignerProvider | Total signers (keys) currently loaded in memory: 298
2023-06-29 21:05:03.874+08:00 | main | INFO  | Runner | Web3Signer has started with TLS disabled, and ready to handle signing requests on 0.0.0.0:9000

The configuration file is as follows

http-cors-origins: ["*"]
http-listen-host: "0.0.0.0"
http-listen-port: 9000
http-host-allowlist: ["*"]

metrics-enabled: true
metrics-host: "0.0.0.0"
metrics-port: 9001
metrics-host-allowlist: "*"

logging: "DEBUG"

swagger-ui-enabled: false

data-path: "/mnt/eth-validatormain/node/web3signer/data"
key-store-path: "/mnt/eth-validatormain/node/web3signer/keys"
eth2.network: mainnet
eth2.key-manager-api-enabled: true
eth2.slashing-protection-enabled: false
eth2.keystores-path: "/mnt/eth-validatormain/node/web3signer/eth2/keystores"
eth2.keystores-passwords-path: "/mnt/eth-validatormain/node/web3signer/eth2/passwords"

[usmansaleem]

@YuXiaoCoder You are probably loading encrypted private keys from local disk, the decryption of these keys at web3signer start up takes time (depending upon the parameters in the encryption file). 300 encrypted keys decrypted within a minute is within expected parameters. The keys loaded from external vaults such as Hashicorp or Azure are already decrypted, i.e. they are stored as decrypted (raw) private key in these vaults. Hope this helps.

Let us know if you are not loading from local encrypted keys but actually attempting to load from Hashicorp vault?

[YuXiaoCoder]

Why do you limit the number of Keys loaded per batch to 10 and the maximum number of CPU cores that can be utilized to 5? https://github.com/Consensys/web3signer/blob/master/signing/src/main/java/tech/pegasys/web3signer/signing/config/SignerLoader.java#L262

[usmansaleem]

@YuXiaoCoder We do limit to max of 5 "availableProcessors" as we have seen performance issues in past when we utilised all the cores.

We are simply reporting 10 keys (an arbitrary number), its not that we are only dealing with 10 keys per batch.

if (filesProcessed % FILES_PROCESSED_TO_REPORT == 0) {
                    LOG.info("{} signing metadata processed", filesProcessed);
}

[YuXiaoCoder]

Is there any local deployment documentation for HashiCorp? Also I didn't find how to configure the use of HashiCorp in the configuration file of Web3Signer, is it by visiting http://127.0.0.1:8200吗? https://docs.web3signer.consensys.net/how-to/store-keys-vaults/hashicorp

[YuXiaoCoder]

How can we improve CPU utilization if we continue to store the Key to local disk

[usmansaleem]

@YuXiaoCoder this loading is a one-time (start up) operation. Once all keys are loaded then web3signer starts signing. If startup of web3signer is a concern, you can use unencrypted keys which loads within seconds as no decryption is involved.

[usmansaleem]

https://docs.web3signer.consensys.net/how-to/store-keys-vaults/hashicorp assumes that you have Hashicorp vault already installed and setup vault locally.

You can check these third party docker compose examples (not endorsed by Consensys) that configures Hashicorp and Web3Signer in dockerized environment. https://github.com/usmansaleem/signers_docker_compose/tree/main/web3signer-hashicorp

[usmansaleem]

Feel free to reach out on our Discord channel web3signer channel for further discussion (https://discord.gg/consensys) or reopen this or a new issue in github repo.