search

dymensionxyz / roller

CLI tool for deploying Dymension RollApps
71 stars 60 forks source link

Invalid Height After a Spike in Usage (panic: invalid height: 7552; expected: 7571) #613

Open nmluthfi opened 9 months ago

nmluthfi commented 9 months ago

Roller version

💈 Roller version v0.1.21-beta 💈 Build time: 2023-11-12T13:35:58+0000" 💈 Git commit: 91828b8725e8bd40831083fd6800a3894e059edc

OS

PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy

Processor Architecture

x86_64

Others

  1. CPU: 6 vCPU Cores
  2. RAM: 16 GB RAM
  3. STORAGE: 400 GB SDD

Steps to reproduce it

I didn't do anything, but one thing that I noticed was that before all of this happened.

My dApps, WorldWide experienced a spike. Around 2k EVM txs happened at the same time. After that, no blocks produces, as you can see in our explorer, the last produced block was block #7654. There is a huge block height different between Portal at 7550 and Explorer at 7564

Expected result

What I want is for my rollapp to become functionally normal. Block height is increasing. image

Actual result

time="2023-12-04T10:14:18+01:00" level=debug msg="Applying block[height 7552 source produced]" module=BlockManager panic: invalid height: 7552; expected: 7571

Relevant log output

time="2023-12-04T10:14:18+01:00" level=info msg="serving HTTP[listen address [::]:26657]"
time="2023-12-04T10:14:18+01:00" level=info msg="listening on[address /ip4/161.97.152.43/tcp/26656/p2p/12D3KooWGHUDgGNCFhBciRj31EuzXv4uabCFiF1kDU9yfP8VcMN9]" module=p2p
time="2023-12-04T10:14:18+01:00" level=info msg="listening on[address /ip4/127.0.0.1/tcp/26656/p2p/12D3KooWGHUDgGNCFhBciRj31EuzXv4uabCFiF1kDU9yfP8VcMN9]" module=p2p
time="2023-12-04T10:14:18+01:00" level=debug msg="setting up gossiping[]" module=p2p
time="2023-12-04T10:14:18+01:00" level=debug msg="setting up DHT[]" module=p2p
time="2023-12-04T10:14:18+01:00" level=info msg="no seed nodes - only listening for connections[]" module=p2p
time="2023-12-04T10:14:18+01:00" level=debug msg="setting up active peer discovery[]" module=p2p
time="2023-12-04T10:14:18+01:00" level=info msg="starting Celestia Data Availability Layer Client[]" module=celestia
time="2023-12-04T10:14:18+01:00" level=debug msg="settlement Layer Client starting.[]" module=settlement_client
time="2023-12-04T10:14:18+01:00" level=info msg="started state updates handler loop[]" module=settlement_client
time="2023-12-04T10:14:18+01:00" level=info msg="Starting the block manager[]" module=BlockManager
time="2023-12-04T10:14:18+01:00" level=info msg="Starting in aggregator mode[]" module=BlockManager
time="2023-12-04T10:14:18+01:00" level=info msg="Started sync target loop[]" module=BlockManager
time="2023-12-04T10:14:18+01:00" level=debug msg="Getting latest batch from settlement layer[]" module=settlement_client
time="2023-12-04T10:14:18+01:00" level=debug msg="Getting latest batch from settlement layer[]" module=settlement_client
time="2023-12-04T10:14:18+01:00" level=info msg="Started retrieve loop[]" module=BlockManager
time="2023-12-04T10:14:18+01:00" level=debug msg="grpc client assigned to client context[target 127.0.0.1:9090]"
time="2023-12-04T10:14:18+01:00" level=info msg="serve[msg Starting RPC HTTP server on [::]:1317]" module=api-server
time="2023-12-04T10:14:18+01:00" level=info msg="Received new syncTarget[syncTarget 7546]" module=BlockManager
time="2023-12-04T10:14:18+01:00" level=info msg="Synced, Starting to produce[current height 7551 syncTarget 7546]" module=BlockManager
time="2023-12-04T10:14:18+01:00" level=info msg="Received new syncTarget[syncTarget 7546]" module=BlockManager
time="2023-12-04T10:14:18+01:00" level=info msg="Synced at height[height 7551]" module=BlockManager
time="2023-12-04T10:14:18+01:00" level=info msg="Using pending block[height 7552]" module=BlockManager
time="2023-12-04T10:14:18+01:00" level=debug msg="Gossiping block[len 527]" module=p2p
time="2023-12-04T10:14:18+01:00" level=debug msg="Applying block[height 7552 source produced]" module=BlockManager
panic: invalid height: 7552; expected: 7571

Notes

I tried to make the rollapp height increase by set the aggregator false in 2 scenarios

  1. Celestia, Sequencer, and Relayer on
  2. Celestia and Sequencer on. Relayer off.

However, none of them works.

This my the current height for both rollapp and hub on Grafana image

molla202 commented 9 months ago

No one found a solution to this. Had to reinstall. sorry Data error caused by resetting the server. or bug entering roles from the server or due to high CPU usage in recent updates. Or maybe there is a problem due to low RAM usage, it was said to be caused by Celestia. In any case, there is only one backup of the network; if the data is corrupted, there is no way to return it.

The most important point of block chain networks is that there is no single copy of the data. but here