taikoxyz / simple-taiko-node

Start your Taiko node with a single command. 🌐
MIT License
1.15k stars 836 forks source link

Bug : [Mainnet] Prover aborted tx send due to critical error: failed to get tx into the mempool #299

Closed quangtuyen88 closed 4 months ago

quangtuyen88 commented 5 months ago

Describe the bug

I'm getting this err :

taiko_client_prover_relayer-1  | ERROR[06-18|02:53:14.580] Submit proof error                       blockID=76054 minTier=200 error="aborted tx send due to critical error: failed to get tx into the mempool"
taiko_client_prover_relayer-1  | ERROR[06-18|02:56:38.646] Submit proof error                       blockID=76054 minTier=200 error="aborted tx send due to critical error: failed to get tx into the mempool"
taiko_client_prover_relayer-1  | ERROR[06-18|03:00:02.711] Submit proof error                       blockID=76054 minTier=200 error="aborted tx send due to critical error: failed to get tx into the mempool"
taiko_client_prover_relayer-1  | ERROR[06-18|03:03:26.777] Submit proof error                       blockID=76054 minTier=200 error="aborted tx send due to critical error: failed to get tx into the mempool"
taiko_client_prover_relayer-1  | ERROR[06-18|03:06:50.842] Submit proof error                       blockID=76054 minTier=200 error="aborted tx send due to critical error: failed to get tx into the mempool"

Log is fine yesterday.

Screenshot 2024-06-18 at 11 03 19

Raiko log is good also

Screenshot 2024-06-18 at 11 03 36

Steps to reproduce

Steps to reproduce here.

Spam policy

quangtuyen88 commented 5 months ago

Screenshot 2024-06-18 at 11 13 41 I can see it's retrying each 3 or 4m . But still fail and it will run out of windows time for submit proof..

If you do retry failed need to reverted that prover transaction. I can't accept lose 250 Taiko again because weird err on mainnet. that's not acceptable.

quangtuyen88 commented 5 months ago

Screenshot 2024-06-18 at 11 29 53 The issue was solved after adding more ETH in prover balance..

  1. We can't monitor our prover 24/24 because that weird err with gas bump.
  2. Team need do action after some retry because 1h windows time for submitting proof.
  3. We need better log clarify on issue so we can have better action. I can't even understand why I'm getting that err.
YoGhurt111 commented 4 months ago

The recommended way is to add a prover balance alert. And will optimize the logger.