search

ArbitrumFoundation / governance

Apache License 2.0
73 stars 27 forks source link

Propmon: retry failing retryables #314

Closed godzillaba closed 1 month ago

godzillaba commented 1 month ago

Trying to execute a retryable that cannot be redeemed will crash propmon. We should probably keep this behavior so we are alerted to this.

However, some retryables are expected to not be redeemable for some period of time. This PR modifies the behavior of RetryableExecutionStage::execute to keep trying to redeem every minute if the retryable ID is in a whitelist. If the retryable is not in the list, propmon will crash as usual.

eg, add this to .env

PENDING_RETRYABLES='["0x42cdcf30aa6de466277d44efa3fef5c0ebd0ca79b2585cb8b79625451dfc4a73","0xbfac911b67e7f79a69e0e9afc329573e3b7a017c1d31e7b368f59cdcab25110a"]'

yahgwai commented 1 month ago

Hmm, ideally propmon wouldnt crash right? We would just want it to log an error for that proposal, but keep monitoring other proposals

For retryables that we expect to fail, maybe we should have it the other way round. Ie keep retrying by default, but then have an override for retryables that we now know will always fail.

godzillaba commented 1 month ago

i removed the whitelist, so it'll just retry everything. i don't think we'll have retryables that never redeem, but we'll have some (like activating the cache manager after stylus activates) that need to wait a bit

so instead of being alerted to a failed retryable via propmon crashing, we'll be notified of the L1 Timelock and failed retryables will be shown in the UI as READY