search

hashgraph / hedera-services

Crypto, token, consensus, file, and smart contract services for the Hedera public ledger
Apache License 2.0
314 stars 138 forks source link

Tune `ZAllocationSpikeTolerance` for ZGC in all networks #4413

Open OlegMazurov opened 1 year ago

OlegMazurov commented 1 year ago

Description

Performance testing shows that under heavy load the memTot metric (the total memory allocated for the heap) hits the specified maximum heap size (currently, 150g), which correlates with allocation stalls reported by ZGC. Increasing ZAllocationSpikeTolerance from the default 1 to 2 has been shown to improve ZGC proactiveness, help keeping the total memory under the maximum size and eliminate allocation stalls. It it recommended to set -XX:ZAllocationSpikeTolerance=2 in all networks.

Steps to reproduce

Observe the memTot metric.

Additional context

No response

Hedera network

mainnet, testnet, previewnet, other

Version

v0.33.0

Operating system

Linux

poulok commented 5 days ago

@OlegMazurov is this ticket still relevant?

OlegMazurov commented 5 days ago

The following files in the repository specify GC flags but do not set -XX:ZAllocationSpikeTolerance:

hedera-node/configuration/testnet/javaOptions
hedera-node/configuration/previewnet/javaOptions
hedera-node/configuration/preprod/javaOptions
hedera-node/configuration/mainnet/javaOptions
hedera-node/test-clients/src/main/resources/testfiles/updateFeature/updateSettings/exec.sh

Either those files should be updated or, if obsolete, deleted from the repository.