search

XRPLF / rippled

Decentralized cryptocurrency blockchain daemon implementing the XRP Ledger protocol in C++
https://xrpl.org
ISC License
4.48k stars 1.45k forks source link

[No sync since upgrade] (Version: 1.8.1) #4041

Open Denis-score opened 2 years ago

Denis-score commented 2 years ago

Hello,

Since I updated my rippled to version 1.8.1 it don't looks to sync anymore to the network.

Here is my server_info : 

   "result" : {
      "info" : {
         "build_version" : "1.8.1",
         "closed_ledger" : {
            "age" : 8,
            "base_fee_xrp" : 1e-05,
            "hash" : "59B9A5A82A7E21AFFBBE23024ED0CB36A1D3AF983EFEB9FF1EF9EF2DD20E6710",
            "reserve_base_xrp" : 200,
            "reserve_inc_xrp" : 50,
            "seq" : 203
         },
         "complete_ledgers" : "empty",
         "fetch_pack" : 14874,
         "hostid" : "xrp-prod-02",
         "io_latency_ms" : 1,
         "jq_trans_overflow" : "0",
         "last_close" : {
            "converge_time_s" : 3.011,
            "proposers" : 0
         },
         "load" : {
            "job_types" : [
               {
                  "in_progress" : 1,
                  "job_type" : "clientCommand"
               },
               {
                  "job_type" : "untrustedValidation",
                  "per_second" : 28
               },
               {
                  "job_type" : "ledgerRequest",
                  "per_second" : 18
               },
               {
                  "in_progress" : 4,
                  "job_type" : "ledgerData",
                  "waiting" : 9
               },
               {
                  "job_type" : "transaction",
                  "per_second" : 10
               },
               {
                  "job_type" : "advanceLedger",
                  "per_second" : 14
               },
               {
                  "job_type" : "fetchTxnData",
                  "peak_time" : 4,
                  "per_second" : 36
               },
               {
                  "job_type" : "trustedValidation",
                  "peak_time" : 1,
                  "per_second" : 6
               },
               {
                  "job_type" : "trustedProposal",
                  "peak_time" : 1,
                  "per_second" : 14
               },
               {
                  "avg_time" : 4,
                  "job_type" : "heartbeat",
                  "peak_time" : 9
               },
               {
                  "job_type" : "peerCommand",
                  "per_second" : 5672
               },
               {
                  "job_type" : "SyncReadNode",
                  "per_second" : 41
               },
               {
                  "job_type" : "AsyncReadNode",
                  "peak_time" : 5,
                  "per_second" : 11384
               },
               {
                  "job_type" : "WriteNode",
                  "peak_time" : 1,
                  "per_second" : 25
               }
            ],
            "threads" : 14
         },
         "load_factor" : 1,
         "network_ledger" : "waiting",
         "node_size" : "huge",
         "peer_disconnects" : "9",
         "peer_disconnects_resources" : "1",
         "peers" : 72,
         "pubkey_node" : "XXXX",
         "pubkey_validator" : "none",
         "published_ledger" : "none",
         "server_state" : "connected",
         "server_state_duration_us" : "887512900",
         "state_accounting" : {
            "connected" : {
               "duration_us" : "888552024",
               "transitions" : 2
            },
            "disconnected" : {
               "duration_us" : "133173491",
               "transitions" : 2
            },
            "full" : {
               "duration_us" : "0",
               "transitions" : 0
            },
            "syncing" : {
               "duration_us" : "0",
               "transitions" : 0
            },
            "tracking" : {
               "duration_us" : "0",
               "transitions" : 0
            }
         },
         "time" : "2021-Dec-16 10:41:49.147203 UTC",
         "uptime" : 1021,
         "validation_quorum" : 28,
         "validator_list" : {
            "count" : 2,
            "expiration" : "2022-May-26 00:00:00.000000000 UTC",
            "status" : "active"
         }
      },
      "status" : "success"
   }
}

And in the logs we have lots of logs like that : 

2021-Dec-16 10:55:06.014658089 UTC Protocol:WRN [875] getLedger: Unable to find ledger
2021-Dec-16 10:55:06.537171379 UTC LedgerConsensus:WRN View of consensus changed during establish status=establish,  mode=wrongLedger
2021-Dec-16 10:55:06.537239050 UTC LedgerConsensus:WRN 0EDED6782294C46F95BC40FFFC16AFB971F4700E5D60A08DA29B886508886084 to 2B610F41BA68488FC77F0DD56C7E29B4E6EA260238AF12F2F5C5A726A0CB5ACC
2021-Dec-16 10:55:06.537344957 UTC LedgerConsensus:WRN {"accepted":true,"account_hash":"AA02C528E37453EBC05CF558A2901E2BE00C780D0B18699B48A39300CC2FF529","close_flags":0,"close_time":692967301,"close_time_human":"2021-Dec-16 10:55:01.000000000 UTC","close_time_resolution":10,"closed":true,"hash":"F4A14A21F2570683D2E140B8E9D79D99A37CB961D4C4A92D12231E2F20F3F465","ledger_hash":"F4A14A21F2570683D2E140B8E9D79D99A37CB961D4C4A92D12231E2F20F3F465","ledger_index":"374","parent_close_time":692967300,"parent_hash":"2B52471650ED17D037E08D49ABEB646459AC6BC457770D3E736EC2502C75D73F","seqNum":"374","totalCoins":"100000000000000000","total_coins":"100000000000000000","transaction_hash":"0000000000000000000000000000000000000000000000000000000000000000"}
2021-Dec-16 10:55:06.538835239 UTC LedgerConsensus:WRN Need consensus ledger 2B610F41BA68488FC77F0DD56C7E29B4E6EA260238AF12F2F5C5A726A0CB5ACC
2021-Dec-16 10:55:07.807701901 UTC LoadMonitor:WRN Job: processLedgerData run: 12237ms wait: 27836ms
2021-Dec-16 10:55:10.754577469 UTC NetworkOPs:WRN We are not running on the consensus ledger
2021-Dec-16 10:55:10.761589502 UTC LedgerConsensus:WRN Need consensus ledger 973D90EE17902C3C1C955A29F897743B5A6AEA89C7A7081E2F9C2CF7054210CE
2021-Dec-16 10:55:11.520696938 UTC LoadMonitor:WRN Job: sweep run: 2729ms wait: 0ms
2021-Dec-16 10:55:12.731999679 UTC LoadMonitor:WRN Job: processLedgerData run: 12199ms wait: 31317ms
2021-Dec-16 10:55:14.692833854 UTC LedgerConsensus:WRN View of consensus changed during establish status=establish,  mode=wrongLedger
2021-Dec-16 10:55:14.692894783 UTC LedgerConsensus:WRN 973D90EE17902C3C1C955A29F897743B5A6AEA89C7A7081E2F9C2CF7054210CE to EB5521543621A86B7702B545CCC8BECC94C38F25169C50991A07E988B0A17585
2021-Dec-16 10:55:14.692952186 UTC LedgerConsensus:WRN {"accepted":true,"account_hash":"D842A8837DF7D9B88F5E37FB9CA67A9162A37510F13E2E1D427D780877577B0A","close_flags":0,"close_time":692967310,"close_time_human":"2021-Dec-16 10:55:10.000000000 UTC","close_time_resolution":10,"closed":true,"hash":"6C5FD39CA668E723513332FB699FCC58D0DFB0BC61B8CC87DFF90C5AB22096BB","ledger_hash":"6C5FD39CA668E723513332FB699FCC58D0DFB0BC61B8CC87DFF90C5AB22096BB","ledger_index":"375","parent_close_time":692967301,"parent_hash":"F4A14A21F2570683D2E140B8E9D79D99A37CB961D4C4A92D12231E2F20F3F465","seqNum":"375","totalCoins":"100000000000000000","total_coins":"100000000000000000","transaction_hash":"0000000000000000000000000000000000000000000000000000000000000000"}
2021-Dec-16 10:55:14.694800244 UTC LedgerConsensus:WRN Need consensus ledger EB5521543621A86B7702B545CCC8BECC94C38F25169C50991A07E988B0A17585
2021-Dec-16 10:55:16.319924011 UTC LoadMonitor:WRN Job: processLedgerData run: 12107ms wait: 34085ms
2021-Dec-16 10:55:17.901370285 UTC LoadMonitor:WRN Job: processLedgerData run: 10094ms wait: 31805ms
sugeladi commented 2 years ago

Has this problem been solved yet?

Rutrus commented 2 years ago

Same problem here. Also happens with 1.8.4. Should I open a new ticket? I created a new issue since the error log is not the same. https://github.com/ripple/rippled/issues/4091

2022-Feb-01 15:53:08.689374863 UTC Application:FTL Result: {}
...
2022-Feb-01 15:53:08.658874684 UTC LedgerMaster:DBG No validated ledger
2022-Feb-01 15:53:08.658866609 UTC ValidatorList:WRN New quorum of 18446744073709551615 exceeds the number of trusted validators (0)
2022-Feb-01 15:53:08.658842513 UTC ValidatorList:DBG 0 of 0 listed validators eligible for inclusion in the trusted set
2022-Feb-01 15:53:08.624412521 UTC ValidatorSite:DBG Loaded 2 sites
cjcobb23 commented 2 years ago

Hi what are your machine specs? Are you using an SSD? An NVME SSD?

Rutrus commented 2 years ago

Today we increased the instance from xlarge. Do you think these values are right?

  instance_type = "r6i.2xlarge"
  ebs_size = "840"
  ebs_type = "gp3"
  ebs_iops = "10000"
cjcobb23 commented 2 years ago

rippled does not run well on EBS. Machines with ephemeral storage do better. The i3 machines work well, such as i3.2xlarge

Rutrus commented 2 years ago

Good to know, I copied your assertion in my (now solved) issue. https://github.com/ripple/rippled/issues/4091

intelliot commented 2 years ago

@Denis-score @sugeladi Can you ensure you're not using EBS?

Let us know if e.g. switching to ephemeral storage resolved your issue. Thanks!