Flaky Test: STAKE_POOLS_JOIN_01rewards, Can join a pool, earn rewards and collect them

[KtorZ]

Context

• 2370 ( https://hydra.iohk.io/build/5250461/nixlog/1/tail )

# Test Case

STAKE_POOLS_JOIN_01rewards

Failure / Counter-example

  src/Test/Integration/Scenario/API/Shelley/StakePools.hs:208:5: 
  1) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_JOIN_01rewards - Can join a pool, earn rewards and collect them
       uncaught exception: RequestException
       DecodeFailure "{\"code\":\"network_unreachable\",\"message\":\"The node backend is unreachable at the moment. Trying again in a bit might work.\"}"

[hasufell]

Is this the same? https://buildkite.com/input-output-hk/cardano-wallet/builds/12747#e17ab417-994b-4af3-9186-09aec0959bc4

      src/Test/Integration/Scenario/API/Shelley/StakePools.hs:808:9:
      1) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_JOIN_01x - Fee boundary values, STAKE_POOLS_JOIN_01x - I can join if I have just the right amount
           uncaught exception: RequestException
           DecodeFailure "{\"code\":\"network_unreachable\",\"message\":\"The node backend is unreachable at the moment. Trying again in a bit might work.\"}"

[rvl]

Is this the same?

It seems likely. I have also had various stake pools tests fail like this locally.

[rvl]

Aside from that error message, this test case is inherently flaky.

If epoch rollover happens just after the rewards withdrawal transaction is submitted, then new rewards accrue before the test case can check that previous rewards were withdrawn.

Symptom of this is:

Waited longer than 90s to resolve action: "Wallet has consumed rewards".
expected: Quantity {getQuantity = 0}
 but got: Quantity {getQuantity = 9007770}

[Anviking]

I see

  5) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_JOIN_01 - Can rejoin another stakepool
       While verifying (Status {statusCode = 200, statusMessage = "OK"},Right (ApiWallet {id = ApiT {getApiT = WalletId {getWalletId = 4139b85eb0c5b1408f4f84e8570818298efde7ea}}, addressPoolGap = ApiT {getApiT = AddressPoolGap {getAddressPoolGap = 20}}, balance = ApiT {getApiT = WalletBalance {available = Quantity {getQuantity = 999998856100}, total = Quantity {getQuantity = 999998856100}, reward = Quantity {getQuantity = 0}}}, delegation = ApiWalletDelegation {active = ApiWalletDelegationNext {status = NotDelegating, target = Nothing, changesAt = Nothing}, next = [ApiWalletDelegationNext {status = Delegating, target = Just (ApiT {getApiT = PoolId {getPoolId = "\236(\243=\203\230\214@\n\RS^3\155\208d|\ts\202l\f\249\194\187\230\131\141\198"}}), changesAt = Just (ApiEpochInfo {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 17}}, epochStartTime = 2021-01-14 17:22:20 UTC})}]}, name = ApiT {getApiT = WalletName {getWalletName = "Faucet Wallet"}}, passphrase = Just (ApiWalletPassphraseInfo {lastUpdatedAt = 2021-01-14 17:21:00.828258 UTC}), state = ApiT {getApiT = Ready}, tip = ApiBlockReference {absoluteSlotNumber = ApiT {getApiT = SlotNo 1535}, slotId = ApiSlotId {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 15}}, slotNumber = ApiT {getApiT = SlotInEpoch {unSlotInEpoch = 35}}}, time = 2021-01-14 17:21:47 UTC, block = ApiBlockInfo {height = Quantity {getQuantity = 692}}}}))
       expected: Just (ApiT {getApiT = EpochNo {unEpochNo = 16}})
        but got: Just (ApiT {getApiT = EpochNo {unEpochNo = 17}})

locally a lot now

[rvl]

@Anviking @piotr-iohk

I see ... [failure] Can rejoin another stakepool ... locally a lot now

I think this new failure may be because I added a waitForNextEpoch command near the end of STAKE_POOLS_JOIN_01rewards to fix a timing-related flakiness.

That changes the start time of the later test case STAKE_POOLS_JOIN_01, which could alter its result.

Then I think these two lines are in the wrong order:

        -- make sure we are at the beginning of new epoch
        (currentEpoch, _) <- getSlotParams ctx
        waitForNextEpoch ctx

and/or the expected value currentEpoch + 3 is incorrect.

[piotr-iohk]

I think this new failure may be because I added a waitForNextEpoch command near the end of STAKE_POOLS_JOIN_01rewards to fix a timing-related flakiness.

I think it shouldn't make a difference actually. :thinking:

Then I think these two lines are in the wrong order:

  -- make sure we are at the beginning of new epoch
   (currentEpoch, _) <- getSlotParams ctx
   waitForNextEpoch ctx

and/or the expected value currentEpoch + 3 is incorrect.

It is a bit awkward, but they are OK I think. It might as well be:

-- make sure we are at the beginning of new epoch
waitForNextEpoch ctx
(currentEpoch, _) <- getSlotParams ctx

and then currentEpoch + 2.

I'm seeing this failure also locally though.

[Anviking]

Btw, I believe I saw it

• Frequently when working on an early version of #2449, locally (potentially due to wrong implementation in that PR)
• Once on something unrelated to #2449, locally
• Not anymore on #2449. Or just too rarely to see it. (however occasionally a WithdrawalsNotInRewardsDELEGS error. Details in that PR)

[Anviking]

Should hopefully be fixed by #2698

Can re-open otherwise.

Note: the issue in the ticket description is technically different from all recently logger errors, but looking past it as it's really old.