search

cardano-foundation / cardano-wallet

HTTP server & command-line for managing UTxOs and HD wallets in Cardano.
Apache License 2.0
763 stars 211 forks source link

Flaky STAKE_POOLS_SMASH_01 - Waited longer than 90s to resolve action: "metadata is fetched" #2331

Open Anviking opened 3 years ago

Anviking commented 3 years ago

Context

Seen once so far, but think the test case is new too. # Test Case

https://github.com/input-output-hk/cardano-wallet/blob/a604358b6c204fe495a082f1c8f0634bdacd5220/lib/core-integration/src/Test/Integration/Scenario/API/Shelley/StakePools.hs#L1101-L1139

Failure / Counter-example

  | src/Test/Integration/Scenario/API/Shelley/StakePools.hs:1119:25:
  | 2) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_SMASH_01 - fetching metadata from SMASH works with delisted pools
  | While verifying (Status {statusCode = 200, statusMessage = "OK"},Right [ApiStakePool {id = ApiT {getApiT = PoolId {getPoolId = "\236(\243=\203\230\214@\n\RS^3\155\208d\|\ts\202l\f\249\194\187\230\131\141\198"}}, metrics = ApiStakePoolMetrics {nonMyopicMemberRewards = Quantity {getQuantity = 14471727896999325}, relativeStake = Quantity {getQuantity = Percentage {getPercentage = 7 % 2000}}, saturation = 1.0487592957486117e-2, producedBlocks = Quantity {getQuantity = 343}}, metadata = Just (ApiT {getApiT = StakePoolMetadata {ticker = StakePoolTicker {unStakePoolTicker = "GPA"}, name = "Genesis Pool A", description = Nothing, homepage = "https://iohk.io"}}), cost = Quantity {getQuantity = 0}, margin = Quantity {getQuantity = Percentage {getPercentage = 1 % 10}}, pledge = Quantity {getQuantity = 2000000000000}, retirement = Nothing, flags = []},ApiStakePool {id = ApiT {getApiT = PoolId {getPoolId = "\187\DC1L\179}u\250\ENQ&\ETX(\194\&5\163\218\226\149\163=\v\166t\165\235\RS>V\142"}}, metrics = ApiStakePoolMetrics {nonMyopicMemberRewards = Quantity {getQuantity = 14471671571815986}, relativeStake = Quantity {getQuantity = Percentage {getPercentage = 23 % 10000}}, saturation = 6.760052819161956e-3, producedBlocks = Quantity {getQuantity = 198}}, metadata = Just (ApiT {getApiT = StakePoolMetadata {ticker = StakePoolTicker {unStakePoolTicker = "GPD"}, name = "Genesis Pool D", description = Just "Lorem Ipsum Dolor Sit Amet.", homepage = "https://iohk.io"}}), cost = Quantity {getQuantity = 0}, margin = Quantity {getQuantity = Percentage {getPercentage = 1 % 10}}, pledge = Quantity {getQuantity = 1000000000000}, retirement = Just (ApiEpochInfo {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 1000000}}, epochStartTime = 2022-02-24 12:29:18 UTC}), flags = []},ApiStakePool {id = ApiT {getApiT = PoolId {getPoolId = "\ESC=\193\156j\184\158\175\252\133\SOH\243u\187\ETX\193\ESC\248\237]\CAN76\177\216\EOT\DC3\214"}}, metrics = ApiStakePoolMetrics {nonMyopicMemberRewards = Quantity {getQuantity = 14471671571815986}, relativeStake = Quantity {getQuantity = Percentage {getPercentage = 23 % 10000}}, saturation = 7.022136204613442e-3, producedBlocks = Quantity {getQuantity = 204}}, metadata = Just (ApiT {getApiT = StakePoolMetadata {ticker = StakePoolTicker {unStakePoolTicker = "GPB"}, name = "Genesis Pool B", description = Nothing, homepage = "https://iohk.io"}}), cost = Quantity {getQuantity = 0}, margin = Quantity {getQuantity = Percentage {getPercentage = 1 % 10}}, pledge = Quantity {getQuantity = 1000000000000}, retirement = Just (ApiEpochInfo {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 100000}}, epochStartTime = 2021-01-03 20:29:18 UTC}), flags = []}])
  | Waited longer than 90s to resolve action: "metadata is fetched".
  | expected: [PoolId {getPoolId = "\180Wh\193\162\218K\209>\188\170\RS\165\DC4\b\237\163\GS\204!v\\\203\212\a\205\169\242"}]
  | but got: []
  |  
  | To rerun use: --match "/API Specifications/SHELLEY_STAKE_POOLS/STAKE_POOLS_SMASH_01 - fetching metadata from SMASH works with delisted pools/"

Resolution

QA

jonathanknowles commented 2 years ago

Seen again in https://hydra.iohk.io/build/14522213.

Log extract:

       Waited longer than 300s to resolve action: "metadata is fetched".
       expected: [PoolId {getPoolId = "\180Wh\193\162\218K\209>\188\170\RS\165\DC4\b\237\163\GS\204!v\\\203\212\a\205\169\242"}]
        but got: []

  To rerun use: --match "/API Specifications/SHELLEY_STAKE_POOLS/STAKE_POOLS_SMASH_01 - fetching metadata from SMASH works with delisted pools/"

Randomized with seed 1544465271

Finished in 1577.2648 seconds, used 683.6826 seconds of CPU time
932 examples, 1 failure, 47 pending
jonathanknowles commented 2 years ago

Seen again: https://github.com/input-output-hk/cardano-wallet/pull/3313#issuecomment-1148333772

Anviking commented 2 years ago

Heinrich suggested that many of the recent failures of SMASH_01 really might have been #3440 in disguise.

This might be right, but it seems like this is also a failure on its own (to what extent only time will tell, after the #3444 fix). This is a failure from https://github.com/input-output-hk/cardano-wallet/pull/3362 before the Babbage cluster setup rework:

  src/Test/Integration/Scenario/API/Shelley/StakePools.hs:1227:25: 
  1) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_SMASH_01 - fetching metadata from SMASH works with delisted pools
       While verifying value:
         ( Status
             { statusCode = 200
             , statusMessage = "OK"
             }
         , Right
             [ ApiStakePool
                 { id = ApiT
                     ( PoolId
                         { getPoolId = "ì(ó=ËæÖ@
                         \x1e^3\x9bÐd|\x9sÊl\xcù»æ\x83\x8dÆ" }
                     )
                 , metrics = ApiStakePoolMetrics
                     { nonMyopicMemberRewards = Quantity 61099330611473
                     , relativeStake = Quantity
                         ( Percentage
                             ( 377 % 1250 )
                         )
                     , saturation = 0.90479930298334
                     , producedBlocks = Quantity 746
                     }
                 , metadata = Just
                     ( ApiT
                         ( StakePoolMetadata
                             { ticker = StakePoolTicker
                                 { unStakePoolTicker = "GPA" }
                             , name = "Genesis Pool A"
                             , description = Nothing
                             , homepage = "https://iohk.io"
                             }
                         )
                     )
                 , cost = Quantity 0
                 , margin = Quantity
                     ( Percentage
                         ( 1 % 10 )
                     )
                 , pledge = Quantity 2000000000000
                 , retirement = Nothing
                 , flags = []
                 }
             , ApiStakePool
                 { id = ApiT
                     ( PoolId
                         { getPoolId = "»\x11L³}uú\x5&\x3(Â5£Úâ\x95£=\xb¦t¥ë\x1e>V\x8e" }
                     )
                 , metrics = ApiStakePoolMetrics
                     { nonMyopicMemberRewards = Quantity 60438778469350
                     , relativeStake = Quantity
                         ( Percentage
                             ( 2201 % 10000 )
                         )
                     , saturation = 0.6604486193611492
                     , producedBlocks = Quantity 549
                     }
                 , metadata = Just
                     ( ApiT
                         ( StakePoolMetadata
                             { ticker = StakePoolTicker
                                 { unStakePoolTicker = "GPD" }
                             , name = "Genesis Pool D"
                             , description = Just "Lorem Ipsum Dolor Sit Amet."
                             , homepage = "https://iohk.io"
                             }
                         )
                     )
                 , cost = Quantity 0
                 , margin = Quantity
                     ( Percentage
                         ( 1 % 10 )
                     )
                 , pledge = Quantity 1000000000000
                 , retirement = Just
                     ( ApiEpochInfo
                         { epochNumber = ApiT
                             ( EpochNo
                                 { unEpochNo = 1000000 }
                             )
                         , epochStartTime = 2023-02-17 18:41:05 UTC
                         }
                     )
                 , flags = []
                 }
             , ApiStakePool
                 { id = ApiT
                     ( PoolId
                         { getPoolId = "´WhÁ¢ÚKÑ>¼ª\x1e¥\x14\x8í£\x1dÌ!v\ËÔ\x7Í©ò" }
                     )
                 , metrics = ApiStakePoolMetrics
                     { nonMyopicMemberRewards = Quantity 54938803886603
                     , relativeStake = Quantity
                         ( Percentage
                             ( 3603 % 10000 )
                         )
                     , saturation = 1.0809649015941327
                     , producedBlocks = Quantity 725
                     }
                 , metadata = Just
                     ( ApiT
                         ( StakePoolMetadata
                             { ticker = StakePoolTicker
                                 { unStakePoolTicker = "GPC" }
                             , name = "Genesis Pool C"
                             , description = Just "Lorem Ipsum Dolor Sit Amet."
                             , homepage = "https://iohk.io"
                             }
                         )
                     )
                 , cost = Quantity 0
                 , margin = Quantity
                     ( Percentage
                         ( 1 % 10 )
                     )
                 , pledge = Quantity 1000000000000
                 , retirement = Just
                     ( ApiEpochInfo
                         { epochNumber = ApiT
                             ( EpochNo
                                 { unEpochNo = 100000 }
                             )
                         , epochStartTime = 2022-07-24 10:41:05 UTC
                         }
                     )
                 , flags = []
                 }
             ]
         )

       Waited longer than 300s to resolve action: "metadata is fetched".
       expected: [PoolId {getPoolId = "\180Wh\193\162\218K\209>\188\170\RS\165\DC4\b\237\163\GS\204!v\\\203\212\a\205\169\242"}]
        but got: []

We see "Genesis Pool C" is there, with metadata and pool id matching "\180Wh\193\162\218K\209>\188\170\RS\165\DC4\b\237\163\GS\204!v\\\203\212\a\205\169\242". The problem is that the pool isn't delisted.

The comment here https://github.com/input-output-hk/cardano-wallet/blob/c047261e9cd552e000951df9dcb72a126b536726/lib/core-integration/src/Test/Integration/Scenario/API/Shelley/StakePools.hs#L1212-L1213 suggests that flakiness previously was reduced by increasing the eventually timeout. Would increasing it again help?

And looking here https://github.com/input-output-hk/cardano-wallet/blob/c6a4dc157d09441d48a013fdd69b0ef8e00a33f0/lib/shelley/src/Cardano/Wallet/Shelley/Pools.hs#L1024 it seems like refetching of the delisted pools can only happen every six hours? The test obviously passes most of the times, but perhaps looking closer into the implementation at some point would help.