Testing service getting stuck with bad sequence error

[ebma]

Sometimes the service gets stuck when retrying after receiving a bad sequence error. It will count up the sequence number indefinitely but never reach a valid one since the number is already too high.

Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404562
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404563
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404564
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404565
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404566
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404567
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404568
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404569
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404570
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404571
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404572
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB
Received a bad sequence error from Horizon, retrying with a new transaction... Used sequence: 210264758245404573
Sending 0.01 USDC to GAH34W7R5EOCVL3Q2PITD6VQ5QASVLY5CFNJ6JQJXR3CYRI7XWF4FT7Y with memo otfNECDFFf5Qb6SkSDuzRMjRNmTB

TODO

We should change the retry logic to fetch the latest valid sequence number from the network in case of failure. For this, we probably just need to remove the caching logic here.

[ebma]

@pendulum-chain/product this should have a rather high priority because it affects the testing service quite often.

[annatekl]

Hey team! Please add your planning poker estimate with Zenhub @b-yap @bogdanS98 @ebma @gianfra-t @TorstenStueber

[gianfra-t]

Quite strange that the sequence number is higher, I don't understand the scenario in which we arrive at that. But I agree that by just removing the cache will solve this since loadAccount will already fetch the the current sequence number.

[annatekl]

@ebma does this require a runtime upgrade?

[ebma]

No it does not.