search

hashgraph / hedera-services

Crypto, token, consensus, file, and smart contract services for the Hedera public ledger
Apache License 2.0
266 stars 119 forks source link

Nightly Canonical Token Transfer tests are failing #13249

Open poulok opened 1 month ago

poulok commented 1 month ago

Nightly canonical token tests are failing because 1 or 3 nodes cannot keep up with transaction handling, causing the secC2C to increase above the acceptable maximum.

Failing test histories

OlegMazurov commented 1 month ago

Running the same benchmark with 100% CPU utilization (all available CPUs) simultaneously on every node in engnet1: engnet1 Node 1 is twice as slow as node 0. The assumption that the network is more or less homogeneous is not valid. The reason for the difference may be poor isolation of VMs in GCP, CPU thermal throttling or something else. The observed behavior - high secC2C on a slow node - is expected under the circumstances (the extent may be questioned, though). From that perspective, the network is not suitable for nightly performance testing.