Closed ebma closed 5 months ago
@pendulum-chain/product this should further reduce the sometimes very long time required for redeem requests.
finally found more info on the nth run:
Apr 01 06:09:16.840 INFO wallet::horizon::responses: interpret_response(): status: 403
Apr 01 06:09:16.840 ERROR wallet::horizon::responses: interpret_response(): got a failure response but failure for returning: wallet::horizon::responses::TransactionResponse
Apr 01 06:09:16.840 ERROR wallet::horizon::responses: interpret_response(): got a failure response but failed to convert to json
Apr 01 06:09:16.840 ERROR wallet::horizon::responses: interpret_response(): got a failure response in string: Ok("error code: 1020")
Nice find 👍 It's a bit weird. I found this section in the docs that mentions that we'd receive a 429
error when exceeding the rate limits. However, this 'error code: 1020' apparently is related to a cloudflare error that is thrown by some firewall rules. So maybe they have two things in place for rate-limiting and we are getting blocked by cloudflare when issuing too many requests. In any case, we should probably consider this a recoverable error so that the vault continues to retry.
Sometimes when the vault client is trying to submit a Stellar transaction as part of redeem requests, the horizon responds with a lot of timeout errors. It seems like at some point, this timeout error changes and the horizon server instead returns a different response that the vault client fails to decode. The vault client then stops trying to execute the redeem request until the next periodic restart.
Example 1: Logs of a Spacewalk client
Example 2: Logs of this failing CI job:
TODO
Make sure that the vault client does not stop retrying the submission of the Stellar transaction for redeem requests.