search

XRPLF / rippled

Decentralized cryptocurrency blockchain daemon implementing the XRP Ledger protocol in C++
https://xrpl.org
ISC License
4.53k stars 1.47k forks source link

InsufficientNetworkMode for hours #3970

Closed daybreaktech closed 3 years ago

daybreaktech commented 3 years ago

Wasn't able to transact into XRPL for hours, throwing errors in Java such org.xrpl.xrpl4j.client.JsonRpcClientErrorException: InsufficientNetworkMode

I noticed that my 'complete_ledgers' has always been 'empty', Set peers to maximum at 20 Node size is medium

Running my machine in Ubuntu Linux 20.04 i5 11th Gen, 6 cores 32GB Ram 1TB Samsung EVO NVME SSD

Restarted and everything but I am getting the same results

2021-Nov-03 10:40:15.148231412 UTC HTTPClient:NFO Connecting to 127.0.0.1:5005

{ "result" : { "info" : { "build_version" : "1.7.3", "closed_ledger" : { "age" : 4, "base_fee_xrp" : 1e-05, "hash" : "B4BA17FA7313644BE5F175A0A6FA16AD405339D238E37B8A0A7C085A0279ECDE", "reserve_base_xrp" : 200, "reserve_inc_xrp" : 50, "seq" : 514 }, "complete_ledgers" : "empty", "fetch_pack" : 89, "hostid" : "daybreaktech", "io_latency_ms" : 1, "jq_trans_overflow" : "0", "last_close" : { "converge_time_s" : 2.001, "proposers" : 0 }, "load" : { "job_types" : [ { "job_type" : "untrustedValidation", "peak_time" : 2, "per_second" : 17 }, { "job_type" : "ledgerRequest", "per_second" : 2 }, { "avg_time" : 1, "job_type" : "ledgerData", "peak_time" : 22, "per_second" : 3 }, { "in_progress" : 1, "job_type" : "clientCommand" }, { "job_type" : "advanceLedger", "per_second" : 11 }, { "job_type" : "fetchTxnData", "per_second" : 43 }, { "job_type" : "trustedValidation", "peak_time" : 1, "per_second" : 4 }, { "job_type" : "trustedProposal", "per_second" : 7 }, { "job_type" : "peerCommand", "per_second" : 197 }, { "job_type" : "SyncReadNode", "per_second" : 120 }, { "job_type" : "AsyncReadNode", "per_second" : 4288 }, { "job_type" : "WriteNode", "per_second" : 114 } ], "threads" : 6 }, "load_factor" : 1, "network_ledger" : "waiting", "peer_disconnects" : "286", "peer_disconnects_resources" : "0", "peers" : 7, "pubkey_node" : "n9KTNP1sz1aknzMjhfgHahULSUuSjwyVNbx3BFSU8FuyWqK9ZGrz", "pubkey_validator" : "none", "published_ledger" : "none", "server_state" : "connected", "server_state_duration_us" : "984913561", "state_accounting" : { "connected" : { "duration_us" : "2773192530", "transitions" : 6 }, "disconnected" : { "duration_us" : "124070288", "transitions" : 6 }, "full" : { "duration_us" : "0", "transitions" : 0 }, "syncing" : { "duration_us" : "0", "transitions" : 0 }, "tracking" : { "duration_us" : "0", "transitions" : 0 } }, "time" : "2021-Nov-03 10:40:15.148634 UTC", "uptime" : 2897, "validation_quorum" : 32, "validator_list" : { "count" : 1, "expiration" : "2022-Jan-13 00:00:00.000000000 UTC", "status" : "active" } }, "status" : "success" } }

ximinez commented 3 years ago

Your node is unable to sync to the network. There are many possible reasons for this. The first thing I'd suggest is to change your node size to "huge", since you probably have sufficient resources, and having more memory available might help.

It's also possible that you don't have sufficient internet connectivity / bandwidth. You have 20 max peers, but this status indicates you only have 7 right now. The possibility is also indicated by the fact that you've got 6 transitions between "disconnected" and "connected", but never even made it into syncing.

It might be worth posting your config files (rippled.cfg and validators.txt), but make sure you remove any secret information, like node_seed or validation_seed.

daybreaktech commented 3 years ago

Your node is unable to sync to the network. There are many possible reasons for this. The first thing I'd suggest is to change your node size to "huge", since you probably have sufficient resources, and having more memory available might help.

It's also possible that you don't have sufficient internet connectivity / bandwidth. You have 20 max peers, but this status indicates you only have 7 right now. The possibility is also indicated by the fact that you've got 6 transitions between "disconnected" and "connected", but never even made it into syncing.

It might be worth posting your config files (rippled.cfg and validators.txt), but make sure you remove any secret information, like node_seed or validation_seed.

Hi, it looks like I don't have node_seed and validation_seed setup

--validators.txt

[validator_list_sites] https://vl.ripple.com

[validator_list_keys] ED2677ABFFD1B33AC6FBC3062B71F1E8397C1505E1C42C64D11AD1B28FF73F4734

----rippled.cfg

[peers_max] 20

[server] port_rpc_admin_local port_peer port_ws_admin_local

port_ws_public

ssl_key = /etc/ssl/private/server.key

ssl_cert = /etc/ssl/certs/server.crt

[port_rpc_admin_local] port = 5005 ip = 127.0.0.1 admin = 127.0.0.1 protocol = http

[port_peer] port = 51235 ip = 0.0.0.0 protocol = peer

[port_ws_admin_local] port = 6006 ip = 127.0.0.1 admin = 127.0.0.1 protocol = ws

[node_size] huge

[node_db] type=NuDB path=/var/lib/rippled/db_10032021/nudb online_delete=512 advisory_delete=0

[database_path] /var/lib/rippled/db_10032021

[debug_logfile] /var/log/rippled/debug.log

[sntp_servers] time.windows.com time.apple.com time.nist.gov pool.ntp.org

[validators_file] validators.txt

[rpc_startup] { "command": "log_level", "severity": "warning" }

[ssl_verify] 1

[ips_fixed] s2.ripple.com 51235

Also, I am getting these timed out errors in logs

2021-Nov-03 15:27:19.836800717 UTC LedgerConsensus:WRN {"accepted":true,"account_hash":"DF2E25107BA921C8B909A198E3B975C12FCA16579B3CA207C694DCDAB881BC4F","close_flags":0,"close_time":689268432,"close_time_human":"2021-Nov-03 15:27:12.000000000 UTC","close_time_resolution":10,"closed":true,"hash":"39376336AF789C8D0902CB99C92B83424C9527A5DD6295BD3102AC779B8551DD","ledger_hash":"39376336AF789C8D0902CB99C92B83424C9527A5DD6295BD3102AC779B8551DD","ledger_index":"105","parent_close_time":689268431,"parent_hash":"D410DDF610425456D4BF9EB23C535D5E2D6EC59597C0EE1F2499D767537B7F43","seqNum":"105","totalCoins":"100000000000000000","total_coins":"100000000000000000","transaction_hash":"0000000000000000000000000000000000000000000000000000000000000000"} 2021-Nov-03 15:27:19.836986076 UTC LedgerConsensus:WRN Need consensus ledger 4C8C59C33BAC3FB930ACDDB5D721DC1A7442E75F8DE089745CD1442437B3A95F 2021-Nov-03 15:27:19.837998715 UTC NetworkOPs:WRN We are not running on the consensus ledger 2021-Nov-03 15:27:21.422452220 UTC InboundLedger:WRN 11 timeouts for ledger 67459928 2021-Nov-03 15:27:21.639526811 UTC InboundLedger:WRN Want: D253B63DDDB1616F163D97D570CC3C29B529FDD8458A981F4AF181AE78FF0ED5 2021-Nov-03 15:27:21.639551651 UTC InboundLedger:WRN Want: E601152FC25DCE43DEA9518A631E4B6C794761CBF42AAD264FFD59FD830FA798 2021-Nov-03 15:27:21.639556286 UTC InboundLedger:WRN Want: EB5B08E9B41EE50E5F543F149B7F802BE23BB3464460DEA0CB022AED21ADFDBD 2021-Nov-03 15:27:21.639560255 UTC InboundLedger:WRN Want: 7FBB65452E7E09E7605A02D0F972D444648428471E4AD235B26F14AD6DDA537C 2021-Nov-03 15:27:21.639563129 UTC InboundLedger:WRN Want: B011988F20992DC54A2A280BF2E231DB30AF540FD3CABCFF807160C54D24F874 2021-Nov-03 15:27:21.639565905 UTC InboundLedger:WRN Want: 47B9012EB0E0BB906A2B30E70919C8F58F9DB44B94D089F611AEF9687A5DB79C 2021-Nov-03 15:27:21.639568784 UTC InboundLedger:WRN Want: 89792DAE1CDD062F2A1D19E5010255CA9CDFD670B637C01BB54926EC1E2815C5 2021-Nov-03 15:27:21.639571590 UTC InboundLedger:WRN Want: D027B24845A857A1DAEADDB4B52A92CCA5D38639CA45060437AEBC33A29074C4 2021-Nov-03 15:27:21.670834392 UTC InboundLedger:WRN Want: 64008EC264A18C4AD48B5962287B8B727F63F080C756A0A0BC8B6921785BA332 2021-Nov-03 15:27:21.670858630 UTC InboundLedger:WRN Want: 7B338BA1DF8EF0E56209DAD1AC2EFF8EC2822E1F6628BD606B01BB984DB602C4 2021-Nov-03 15:27:21.670862368 UTC InboundLedger:WRN Want: 8B7C39B5F6DB74A3C0584F547124A009835696ED9E8AEF991CCDD243B0CB0D2A 2021-Nov-03 15:27:21.670865304 UTC InboundLedger:WRN Want: E7842F0CE5332253B36D90E8044B8C8B2A67DD55CF21A98B6F7655902B829373 2021-Nov-03 15:27:21.670876257 UTC InboundLedger:WRN Want: B3AC46DAC22659436985D66828BA25B6AC4E05A2567117F1A5D8790BDA7F7544 2021-Nov-03 15:27:21.670879326 UTC InboundLedger:WRN Want: C000641312423860342599191A83E22C01E16113CDA76BB2E3122D911099AB05 2021-Nov-03 15:27:21.670882149 UTC InboundLedger:WRN Want: 3FAFD1259D61A958DEEEB8C9A504BD11238DEA273D0E7095EFDC4F212093D802 2021-Nov-03 15:27:21.670884995 UTC InboundLedger:WRN Want: 73FDACE245C303F28ACAFDB2578DEFCD06E5BDCBD97D056D05E805FA173BC78F 2021-Nov-03 15:27:22.084592417 UTC InboundLedger:WRN Want: F291667859C7F78471DB4D889CB82B96B0173A77B230766829B9368B753F8E4A 2021-Nov-03 15:27:22.084669143 UTC InboundLedger:WRN Want: F67716622FF423FED7C3D5A149F0E36C24B81F887C0926781057BA75650E3EA6 2021-Nov-03 15:27:22.084688095 UTC InboundLedger:WRN Want: 3B43E45881C59114AF95B280AC2EC6F9A12858B76FA8CF5D425B54FACB52DAEE 2021-Nov-03 15:27:22.084703052 UTC InboundLedger:WRN Want: 3F4AA63B1DA5550D141D6C65534530177648331C3A97686CD37F7AA222CC8142 2021-Nov-03 15:27:22.084721256 UTC InboundLedger:WRN Want: F2ED10D830C10FB44DADC80E21EF754789BF95D16104D4831972F09EA6040C3D 2021-Nov-03 15:27:22.084736010 UTC InboundLedger:WRN Want: FBA3774A0CF7211939CD814D2993F373517C5C65ED5E75149A7E64F4CE0B24F6 2021-Nov-03 15:27:22.084754089 UTC InboundLedger:WRN Want: FE8E37BF5F661E973744D09F6DAA1D40EEB5D86FB3243DCEDAF4C94B1F0B1DDE 2021-Nov-03 15:27:22.084768700 UTC InboundLedger:WRN Want: 17927B1574811C7BCDEE7712D103CF2A5E060B94FD85334374D5605B6EF4DC69 2021-Nov-03 15:27:22.787194036 UTC InboundLedger:WRN 11 timeouts for ledger 67459930 2021-Nov-03 15:27:22.812079270 UTC InboundLedger:WRN Want: D308C86D0C4EC2A14C79813A4E87F9682533992553051E9756AF6D6FC69963D3 2021-Nov-03 15:27:22.812188675 UTC InboundLedger:WRN Want: FA634E7BB56403F88D1887DF5BCB6C46F149F91B2D60DD5C522E12F0EF72D870 2021-Nov-03 15:27:22.812229473 UTC InboundLedger:WRN Want: E636DAA13ABCAA51E6A98C39FC6DEAEC8D48FD018835FF07D5C6A047674AFE4F 2021-Nov-03 15:27:22.812266033 UTC InboundLedger:WRN Want: D3EA155B1FFB46C82D66765D8982E1156C03E878808F3DB30DA1DDF09CA499A0 2021-Nov-03 15:27:22.812307299 UTC InboundLedger:WRN Want: FB6B7D137871BD887465718BEA5D296A38F33F51EE993DC33F5D72062FEB43A6 2021-Nov-03 15:27:22.812354771 UTC InboundLedger:WRN Want: 24D6AB0D21526EA12D035757F7180B9A4C015055E43485734097176A374FB98F 2021-Nov-03 15:27:22.812388387 UTC InboundLedger:WRN Want: 208FE5815475C795ECEC549AE1A6EE35DF11301790640D3A37B4E334FA18FCB1 2021-Nov-03 15:27:22.812421037 UTC InboundLedger:WRN Want: 311F00FDAE7B531FC5DB8C5BC0D9B047BDDBCC2A792E2F42FE4C70A3FBDD4528 2021-Nov-03 15:27:22.840502195 UTC NetworkOPs:WRN We are not running on the consensus ledger 2021-Nov-03 15:27:23.839740763 UTC LedgerConsensus:WRN View of consensus changed during open status=open, mode=wrongLedger 2021-Nov-03 15:27:23.839821507 UTC LedgerConsensus:WRN 4C8C59C33BAC3FB930ACDDB5D721DC1A7442E75F8DE089745CD1442437B3A95F to 9D159AC6B7DC4D20C2AC1B88E0239F647499E227ABC12E2B55BAAF81B4C86A75 2021-Nov-03 15:27:23.839921132 UTC LedgerConsensus:WRN {"accepted":true,"account_hash":"BBF352AF1DCCB2C3D4997999E878C979457478062C27C990E66B5622D0B4AC51","close_flags":0,"close_time":689268441,"close_time_human":"2021-Nov-03 15:27:21.000000000 UTC","close_time_resolution":10,"closed":true,"hash":"47907A5E59B2403923228A625AB93D1A406070BED1AC0A103170369BBC9CCFB5","ledger_hash":"47907A5E59B2403923228A625AB93D1A406070BED1AC0A103170369BBC9CCFB5","ledger_index":"107","parent_close_time":689268440,"parent_hash":"8F1677F7394FF5EF43108B691563A0C24152080B9DF26ACF7CB4918F64039C22","seqNum":"107","totalCoins":"100000000000000000","total_coins":"100000000000000000","transaction_hash":"0000000000000000000000000000000000000000000000000000000000000000"} 2021-Nov-03 15:27:23.840058174 UTC LedgerConsensus:WRN Need consensus ledger 9D159AC6B7DC4D20C2AC1B88E0239F647499E227ABC12E2B55BAAF81B4C86A75 2021-Nov-03 15:27:23.859279740 UTC TransactionAcquire:WRN Failed to acquire TX set C8295FD17AD3F223B5DBDBD23CA96D66C0F0A68609E3638DD3FDC56FD5D21408 2021-Nov-03 15:27:24.023277757 UTC TransactionAcquire:WRN Failed to acquire TX set 6092972EE23D9B3FB309A1CCADC67C3C6A76B2EF0BF651E1FACD5A89FFBB33D6 2021-Nov-03 15:27:24.023496585 UTC TransactionAcquire:WRN Failed to acquire TX set 645EC8FB1F9A63BB045DB1539D16028346051549B5C08200E84578D0241EE083 2021-Nov-03 15:27:24.110425397 UTC TransactionAcquire:WRN Failed to acquire TX set 25B5ED46F7BD76907FAD277224CDF0CAC0374154A2FA97293F0999FFF39B4E40 2021-Nov-03 15:27:24.110526837 UTC TransactionAcquire:WRN Failed to acquire TX set A097B223C7E2912AB4FFD2D7FA3CACB79960FE4DEC85A5CB8C7E3F4587D625E6 2021-Nov-03 15:27:24.171292087 UTC InboundLedger:WRN Want: 1C1A6F3AA90C1A50B244A5058CF2AF039542388127F7CB02FFB4665E92ADE07B 2021-Nov-03 15:27:24.171315951 UTC InboundLedger:WRN Want: 9D0B2192DB5BE3CA0ABEF8F84BD4A8F3D1231E532DFB6F4DDD58AFB516540DA6 2021-Nov-03 15:27:24.171320489 UTC InboundLedger:WRN Want: 146265D4C1F93D86B405089D387D6E1EF8896F06D8D5CA2B137C67389C839B3F 2021-Nov-03 15:27:24.171323414 UTC InboundLedger:WRN Want: 96EFFD3C8EB5D88C7374CB2EEB9CD623ECDE6A20CCF2220DC6A9DBB8D1BDD9B6 2021-Nov-03 15:27:24.171326692 UTC InboundLedger:WRN Want: B3AC46DAC22659436985D66828BA25B6AC4E05A2567117F1A5D8790BDA7F7544 2021-Nov-03 15:27:24.171330043 UTC InboundLedger:WRN Want: 73FDACE245C303F28ACAFDB2578DEFCD06E5BDCBD97D056D05E805FA173BC78F 2021-Nov-03 15:27:24.171333851 UTC InboundLedger:WRN Want: C000641312423860342599191A83E22C01E16113CDA76BB2E3122D911099AB05 2021-Nov-03 15:27:24.171338354 UTC InboundLedger:WRN Want: 3FAFD1259D61A958DEEEB8C9A504BD11238DEA273D0E7095EFDC4F212093D802 2021-Nov-03 15:27:24.174179061 UTC InboundLedger:WRN Want: D253B63DDDB1616F163D97D570CC3C29B529FDD8458A981F4AF181AE78FF0ED5 2021-Nov-03 15:27:24.174193954 UTC InboundLedger:WRN Want: E601152FC25DCE43DEA9518A631E4B6C794761CBF42AAD264FFD59FD830FA798 2021-Nov-03 15:27:24.174198100 UTC InboundLedger:WRN Want: EB5B08E9B41EE50E5F543F149B7F802BE23BB3464460DEA0CB022AED21ADFDBD 2021-Nov-03 15:27:24.174201350 UTC InboundLedger:WRN Want: 7FBB65452E7E09E7605A02D0F972D444648428471E4AD235B26F14AD6DDA537C 2021-Nov-03 15:27:24.174204336 UTC InboundLedger:WRN Want: 84E3457E12B6F743F303B064C40CA81B982DF2159B0AC6BA29D86A4747DBDC1B 2021-Nov-03 15:27:24.174207306 UTC InboundLedger:WRN Want: 9F8C6C005D7EF0D88BBF886FEBC7F729FA7847003351880D0DE152A1A2E6E78A 2021-Nov-03 15:27:24.174211091 UTC InboundLedger:WRN Want: 2ED98F776D82854C48045D49E4C250297076EBE4090D4B19D4F24B4CA1CC3B7D 2021-Nov-03 15:27:24.174214021 UTC InboundLedger:WRN Want: 02911CB1E6869A48CA8AEA1F857912B196B82DEF8F6C86638710D8DE7CF40808 2021-Nov-03 15:27:24.205411706 UTC TransactionAcquire:WRN Failed to acquire TX set 91DE41C27DF76343314793E92AB73EE7D2C1533673C2A84F798E731D3F3C8B16 2021-Nov-03 15:27:24.205974525 UTC TransactionAcquire:WRN Failed to acquire TX set F16A636D717AFE64DC890085CD30F54A6852052065E049F8D803B633ECBBE84B 2021-Nov-03 15:27:24.206328881 UTC TransactionAcquire:WRN Failed to acquire TX set D571FE516B127AD4CD9F3C33F8ABF99A93E68B50540E4DECCDA12AE2872A3E8B 2021-Nov-03 15:27:24.237450102 UTC TransactionAcquire:WRN Failed to acquire TX set 1E443AFF944483C089E9EB6C317FB8211E31C7BC437B1CB98BB480E12EC1869C 2021-Nov-03 15:27:24.256574754 UTC TransactionAcquire:WRN Failed to acquire TX set D1AD8A82010E12B1C4CB4415F772AFC7CAFF303C35F37917C9F5D4A9163FE439 2021-Nov-03 15:27:24.273372713 UTC TransactionAcquire:WRN Failed to acquire TX set 6EB24A45264483EF38D828E0F14E9050748CCECB9C9D6A993CAD1C733C611480 2021-Nov-03 15:27:24.273828146 UTC TransactionAcquire:WRN Failed to acquire TX set 6247A8F821EBF15EB3CED0D7CF33EC2CFF430149936819A1AAFE7F79D7556480 2021-Nov-03 15:27:24.275256692 UTC TransactionAcquire:WRN Failed to acquire TX set E0AB29BA3B5C62F24A0A9F91F18F3DCD6F87C163462F3E2C114EDAC797EC5A5B

daybreaktech commented 3 years ago

Nevermind, I changed router and my internet improve from 10MBPS to 50MBPS and it starting working back again.

ximinez commented 3 years ago

I happy to hear it's working now!