search

XRPLF / rippled

Decentralized cryptocurrency blockchain daemon implementing the XRP Ledger protocol in C++
https://xrpl.org
ISC License
4.5k stars 1.45k forks source link

Slow to restart for a full-history node #3227

Closed luleigreat closed 1 year ago

luleigreat commented 4 years ago

I have a private ripple-network,and have ran for about two years with 5 full-history valiator nodes.

Recently,when I restart a full-history node, it takes so long time to begin work（about 24 hours) and even some nodes will not work(complete_ledgers is always "empty").

Does anybody know why? Is this a normal phenomenon? And how can I start a full-history node and let it work quickly.

The debug.log shows below:

2020-Jan-15 03:18:58 LedgerConsensus:WRN Need consensus ledger 1849022A6743A07BD1B5FCC0CAAA539D71718BCDF5F38E63DEE97B26D2E15AC4 2020-Jan-15 03:19:01 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:01 LedgerConsensus:WRN Need consensus ledger B4D6966EBB97337E288692545EC88050A15E42A888C69AFB41A44E42228DF0A4 2020-Jan-15 03:19:04 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:06 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:06 LedgerConsensus:WRN Need consensus ledger 91C933ED19EA3EA5689BA8EF3CDCAB149DBE471D0E0C2EB2AE5001C16AC4F46A 2020-Jan-15 03:19:08 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:08 LedgerConsensus:WRN Need consensus ledger 210E563AF58EDDB6B5F7733DC59157F36C66B86100CC8CA94AA5D2CBABDEEDAB 2020-Jan-15 03:19:11 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:11 LedgerConsensus:WRN Need consensus ledger 0BD9D507F881900FC4BF23D8EC6989D330DD07DBBD54B76F88658F354C64E083 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 80679ms wait: 85801ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17576ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17575ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17575ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17573ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17568ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17568ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17568ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17567ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17567ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17566ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17566ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17563ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17560ms 2020-Jan-15 03:19:13 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 17559ms 2020-Jan-15 03:19:14 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:14 LedgerConsensus:WRN Need consensus ledger 8CEC221976D93011926B636B0538A5B7A9549EA2984D8AED07374F5946739DAB 2020-Jan-15 03:19:18 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:18 LedgerConsensus:WRN Need consensus ledger 206A5B7D2639CB1034B54871C238F486DD791DE99D27A9B028B97C8DE3BB8E06 2020-Jan-15 03:19:21 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:21 LedgerConsensus:WRN Need consensus ledger 8D0B1B78280BA710EB69EB38C9FBF43EC07EA1CD7B64D59C135DF88993C12B3E 2020-Jan-15 03:19:24 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:24 LedgerConsensus:WRN Need consensus ledger C49E82173C569742599B7C10D28D5CA15538C9CF85E1BFB236FDDDAB86650BBD 2020-Jan-15 03:19:27 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:27 LedgerConsensus:WRN Need consensus ledger F1C6F5ECCE3B45FA768EEC4E1EB3D2B2FBCFA897703E3A0E2961684EF161705E 2020-Jan-15 03:19:30 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:30 LedgerConsensus:WRN Need consensus ledger 1D60FB534AC13B74FF6332BDF94DC62E97E4C5FD7BD59B30B708BE07633F94BD 2020-Jan-15 03:19:33 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:33 LedgerConsensus:WRN Need consensus ledger E4C1774DB5B8B5BA3162EDF35EF83153483D27AF9516ACD545AE06FBB7915222 2020-Jan-15 03:19:36 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:38 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:38 LedgerConsensus:WRN Need consensus ledger 333FCB2A965235A0DFFD0A0DE807EBE2E1CD3C0F458BD43DDF2515AC166F32B5 2020-Jan-15 03:19:40 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:40 LedgerConsensus:WRN Need consensus ledger 9AA97CFABFF62E722C78D2DFFC7AB5711FC493F0D14014F51F859FBDDDB127B6 2020-Jan-15 03:19:43 NetworkOPs:WRN We are not running on the consensus ledger 2020-Jan-15 03:19:43 LedgerConsensus:WRN Need consensus ledger FE362B0ECFAC6DD679EEDF6580E66307C6AEF9198672A918D3E518744C6F44E5

nbougalis commented 4 years ago

The server should start fast, regardless of the amount of history. I don't think your problem is related to the amount of history.

Please note that 1.4 made some changes that may cause some servers to perform time-consuming house-keeping; this happens only once, the first time you upgrade to 1.4. During this, the server appears unresponsive. Could this be the issue you are experiencing?

luleigreat commented 4 years ago

No, not the 1.4 version issue, I am using rippled version 0.80.2. The server will start fast without any history (It's fast when I change the db directory to another blank directory),but start very slow when using the full history.

nbougalis commented 4 years ago

Some loss of speed is reasonable when the server has to process historical data, but it should not affect the server as severely as you suggest given then syncing algorithm.

ksemaev commented 4 years ago

@nbougalis I have my ripple full history nodes starting up forever after restart. I notices that it took 40 min for one of my node to start after upgrade, and it takes about 30 hours for 2 others. Is there any way to see if they indeed do house-keeping or they are stuck? Some cli commands, or api calls maybe? Will appreciate a link to any guide/article describing this first start after upgrade procedures

miguelportilla commented 4 years ago

@ksemaev If you are using version 1.40, please read the following upgrade advisory. https://xrpl.org/blog/2020/rippled-1.4.0-upgrade-advisory.html

intelliot commented 1 year ago

Closing due to inactivity. If the issue persists, please open a new issue.