search

hashgraph / hedera-services

Crypto, token, consensus, file, and smart contract services for the Hedera public ledger
Apache License 2.0
313 stars 138 forks source link

Fix the flaky DAB enabled upgrade test #16335

Open Neeharika-Sompalli opened 3 weeks ago

Neeharika-Sompalli commented 3 weeks ago

Sometimes the DAB upgrade test fails with this error, and on re-run it passes. [Example run ] (https://github.com/hashgraph/hedera-services/actions/runs/11608640117/job/32324573024?pr=15930)

  Upgrading with DAB enabled > after removing node id1 > exports an address book without node1 and pays its stake no rewards ✘ as written (1m 10s)
    java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.IllegalStateException: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException
    Caused by: java.lang.IllegalStateException: java.lang.IllegalStateException: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException
    Caused by: java.lang.IllegalStateException: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException
    Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException
    Caused by: java.util.concurrent.TimeoutException

This test should be fixed to avoid flakiness

iwsimon commented 3 weeks ago

Just ran DabEnabledUpgradeTest in develop branch. Tests passed 12. But with some io exception: 2024-10-31 17:06:11.864 ERROR 202 HapiTxnOp - body=transactionID { transactionValidStart { seconds: 1730408711 nanos: 1944 } accountID { accountNum: 2 } } nodeAccountID { accountNum: 3 } transactionFee: 84010 transactionValidDuration { seconds: 120 } memo: "\303\203\302\256\303\202\302\267\303\203\302\271tF8\303\202\302\256J\303\203\302\213\303\203\302\220\303\203\302\216" cryptoTransfer { transfers { accountAmounts { accountID { accountNum: 800 } amount: 1 } accountAmounts { accountID { accountNum: 2 } amount: -1 } } }; sigs=sigPair { pubKeyPrefix: "\n" ed25519: "\204\0344_\345X\262V\202\223\210\nq\213im>&\331\372\252\333.\000}\256\345\036\206E\343\'\332\f\276I=\022C\222\022\314\361\326\224{,\t9\252\244\371\213\307\200\0004\017\267O\347\031q\f" } Status resolution failed due to unrecoverable runtime exception, possibly network connection lost. io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268) ~[grpc-stub-1.65.1-module.jar:1.65.1] at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249) ~[grpc-stub-1.65.1-module.jar:1.65.1] at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167) ~[grpc-stub-1.65.1-module.jar:1.65.1] at com.hederahashgraph.service.proto.java.CryptoServiceGrpc$CryptoServiceBlockingStub.cryptoTransfer(CryptoServiceGrpc.java:1029) ~[hapi-0.56.0.jar:?] at com.hedera.services.bdd.junit.hedera.utils.GrpcUtils.submit(GrpcUtils.java:129) ~[main/:?] at com.hedera.services.bdd.junit.hedera.AbstractGrpcNetwork.submit(AbstractGrpcNetwork.java:84) ~[main/:?] at com.hedera.services.bdd.spec.transactions.HapiTxnOp.lambda$static$0(HapiTxnOp.java:90) ~[main/:?] at com.hedera.services.bdd.spec.transactions.HapiTxnOp.submitOp(HapiTxnOp.java:193) ~[main/:?] at com.hedera.services.bdd.spec.HapiSpecOperation.execFor(HapiSpecOperation.java:244) ~[main/:?] at com.hedera.services.bdd.spec.utilops.CustomSpecAssert.handleExec(CustomSpecAssert.java:63) ~[main/:?] at com.hedera.services.bdd.spec.utilops.CustomSpecAssert.executeHederaOps(CustomSpecAssert.java:46) ~[main/:?] at com.hedera.services.bdd.spec.utilops.CustomSpecAssert.allRunFor(CustomSpecAssert.java:40) ~[main/:?] at com.hedera.services.bdd.spec.utilops.CustomSpecAssert.allRunFor(CustomSpecAssert.java:71) ~[main/:?] at com.hedera.services.bdd.spec.utilops.UtilVerbs.lambda$runBackgroundTrafficUntilFreezeComplete$5(UtilVerbs.java:547) ~[main/:?] at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) [?:?] at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1796) [?:?] at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) [?:?] at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312) [?:?] at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843) [?:?] at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808) [?:?] at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188) [?:?]