search

sedaprotocol / seda-chain

SEDA chain network
GNU General Public License v3.0
18 stars 9 forks source link

🐛 Block production sometimes takes minutes #356

Open Thomasvdam opened 2 months ago

Thomasvdam commented 2 months ago

🐛 Bug Report

We notice that on a few occasions the production/finalising of a block took way more than the ~7 seconds we expect.

Steps to Reproduce

Unknown, but it can be observed on Devnet. We suspect tally VM execution to play a role.

Code snippet to reproduce

N.A.

Stack trace & error message

N.A.

Expected Behavior

Block production time should be relatively consistent, especially when the load is consistent.

Your Environment

Thomasvdam commented 2 months ago

We suspect the timeout in the tally vm might cause/contribute to the issues we're observing.

Thomasvdam commented 1 month ago

With the deployed debug builds from Friday September 13th we have not observed block lag over the weekend despite a data request being issued and resolved every 10 minutes. We do see that memory usage is steadily climbing in both nodes

Screenshot 2024-09-16 at 15 02 05 (the last spike is due to to block lag check script)

Thomasvdam commented 1 month ago

Keeping notes in our Notion: https://www.notion.so/sedaprotocol/2024-09-Block-Production-Lag-107a68d575ca801c9673d1c08824c901

Thomasvdam commented 1 month ago

Got the durations between all SIGABRTs for the past 10 days, rounded to make it a little easier to digest:

{
  "<1 hour": 13,
  "1 hour": 29,
  "2 hours": 24,
  "3 hours": 23,
  "4 hours": 19,
  "5 hours": 3
}