Closed brianoflondon closed 1 year ago
@agates
I had to do a fair bit of work and testing to make RC Depletion handle as a retry. I believe I have all the automated tests working though I will admit they sometimes fail on Main net even though I know they work more reliably on the post HF26 FakeNet (this is desired anyway).
In the end very small code changes.
It's quite important that @daveajones starts running this today I think.
I know we've cut this really tight.
A key point is that I've hard coded the 3 most reliable API servers in. I think this is necessary for the next week or so AT LEAST as I know these three should be reliable. After everything shakes out we might go back to the default list in Lighthive.
else:
nodes = [
"https://hive-api.arcange.eu",
"https://api.deathwing.me",
"https://api.hive.blog",
]
This is from the automated test and I'll explain below:
2022-10-11 08:34:25 [ ERROR] Failed to send 7 IRIs (podping_hivewriter.py:489)
2022-10-11 08:34:25 [ ERROR] (podping_hivewriter.py:492)
2022-10-11 08:34:25 [ WARNING] Malformed error response (podping_hivewriter.py:506)
2022-10-11 08:34:25 [ INFO] FAILURE COUNT: 47 - RETRYING 7 IRIs (podping_hivewriter.py:473)
2022-10-11 08:34:27 [ ERROR] send_notification error: payer has not enough RC mana for transaction:Account: *** has 271970563 RC, needs 979072257 RC. Please wait to transact, or power up HIVE. (podping_hivewriter.py:404)
2022-10-11 08:34:27 [ ERROR] payer has not enough RC mana for transaction:Account: *** has 271970563 RC, needs 979072257 RC. Please wait to transact, or power up HIVE. (podping_hivewriter.py:415)
2022-10-11 08:34:27 [ ERROR] Failed to send 7 IRIs (podping_hivewriter.py:489)
2022-10-11 08:34:27 [ ERROR] (podping_hivewriter.py:492)
2022-10-11 08:34:27 [ WARNING] Malformed error response (podping_hivewriter.py:506)
2022-10-11 08:34:27 [ INFO] FAILURE COUNT: 48 - RETRYING 7 IRIs (podping_hivewriter.py:473)
2022-10-11 08:34:29 [ ERROR] send_notification error: payer has not enough RC mana for transaction:Account: *** has 271970563 RC, needs 979072257 RC. Please wait to transact, or power up HIVE. (podping_hivewriter.py:404)
2022-10-11 08:34:29 [ ERROR] payer has not enough RC mana for transaction:Account: *** has 271970563 RC, needs 979072257 RC. Please wait to transact, or power up HIVE. (podping_hivewriter.py:415)
2022-10-11 08:34:29 [ ERROR] Failed to send 7 IRIs (podping_hivewriter.py:489)
2022-10-11 08:34:29 [ ERROR] (podping_hivewriter.py:492)
2022-10-11 08:34:29 [ WARNING] Malformed error response (podping_hivewriter.py:506)
2022-10-11 08:34:29 [ INFO] FAILURE COUNT: 49 - RETRYING 7 IRIs (podping_hivewriter.py:473)
2022-10-11 08:34:30 [ INFO] Lighthive Node: https://api.hive.blog/ (podping_hivewriter.py:401)
2022-10-11 08:34:30 [ INFO] JSON size: 682 (podping_hivewriter.py:402)
2022-10-11 08:34:30 [ INFO] FAILURE CLEARED after 49 retries (podping_hivewriter.py:486)
2022-10-11 08:34:30 [ INFO] Batch send time: 101.56 | Failures: 49 - IRI batch_id 4636b5e0-349d-4b70-9a85-dde7ad26bbb8 | IRIs in batch: 7 | last_node: https://api.hive.blog/ (podping_hivewriter.py:236)
PASSED
During the git test I actually managed to run out of Resource Credits for the podping.gittest
account. This happened TODAY because the RC Delegation bot I'm running doesn't yet work on Main Net so while this test was stalled, I manually gave more Hive Power to the podping.gittest
account. That did the trick and after 49 retries the test continued successfully.
Post HardFork 26 this RC Delegation (instead of a Hive Power delegation) would have happened automatically.
I actually want to change the messages here but right now I'm sure it is working and I'm going to leave it alone. Testing against the HF25 and HF26 chains simultaneously is really hard so I'd rather cosmetically improve this after everything shakes out.
As far as writing test scripts to simulate this I think it is not necessary right now but perhaps I will think about how to do that.
We used to have in the code a facility to switch to a testnet API but that was removed.
I'm looking at either putting that back so I can run tests for the upcoming Hard Fork:
https://peakd.com/hive-139531/@blocktrades/14th-update-of-2022-hive-hf26-release-candidate-and-mirrornet-api-node-ready-for-app-testing