search

XRPLF / rippled

Decentralized cryptocurrency blockchain daemon implementing the XRP Ledger protocol in C++
https://xrpl.org
ISC License
4.53k stars 1.47k forks source link

rippled stops sync with "Unable to determine hash of ancestor" error #2746

Open lcgogo opened 6 years ago

lcgogo commented 6 years ago

The ripple 1.1.0 stop sync and the logs shows followed error:

2018-Nov-01 04:34:48.774969499 Validations:WRN Unable to determine hash of ancestor seq=42640542 from ledger hash=FB5821F5BF23124CB7F7C0409923BE89A073711CE8909BDDD76FFB06E94FD452 seq=42642925 2018-Nov-01 04:34:49.775236865 Validations:WRN Unable to determine hash of ancestor seq=42640542 from ledger hash=FB5821F5BF23124CB7F7C0409923BE89A073711CE8909BDDD76FFB06E94FD452 seq=42642925 ^[2018-Nov-01 04:34:50.775574406 Validations:WRN Unable to determine hash of ancestor seq=42640542 from ledger hash=FB5821F5BF23124CB7F7C0409923BE89A073711CE8909BDDD76FFB06E94FD452 seq=42642925

root@node-03:/data/xrp/xrp# curl -H "Content-Type: application/json" -X POST -d '{"method":"server_info"}' http://172.16.254.3:11001 {"result":{"info":{"build_version":"1.1.0","complete_ledgers":"42434953-42640545","hostid":"170bbd918c6d","io_latency_ms":1,"jq_trans_overflow":"0","last_close":{"converge_time_s":2,"proposers":0},"load":{"job_types":[{"in_progress":1,"job_type":"clientCommand"}],"threads":4},"load_factor":1,"peer_disconnects":"14902","peer_disconnects_resources":"0","peers":0,"pubkey_node":"n94Nt6QxdtBsmeERmEw4PMEF2HuTsJ5sjgdqpebBSaj7tah1at6K","pubkey_validator":"none","server_state":"full","state_accounting":{"connected":{"duration_us":"353863624","transitions":2},"disconnected":{"duration_us":"1407149","transitions":1},"full":{"duration_us":"222800652720","transitions":173},"syncing":{"duration_us":"490046235","transitions":174},"tracking":{"duration_us":"38390","transitions":173}},"time":"2018-Nov-01 04:40:50.864363","uptime":223646,"validated_ledger":{"age":47840,"base_fee_xrp":1e-05,"hash":"99A24C85EBB39989AD5BA0A870778E2FCBD5A5DCBCFAD92B642EA15C72E77BB2","reserve_base_xrp":20,"reserve_inc_xrp":5,"seq":42640545},"validation_quorum":19,"validator_list_expires":"2018-Nov-07 00:00:00.000000000"},"status":"success"}}

nbougalis commented 6 years ago

Your server has no peers (no connections) and hasn’t been able to track the network for a while.

Do you have debug logging enabled?

woshidama323 commented 6 years ago

same issue found some log in /var/log/rippled/debug.log on centos7 2018-Nov-19 08:27:32.538739853 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=1902B6EE20BC19A14744A01F1AB6ADDAF5E3B95A905EAE9F5C179D6A5D07784B seq=43092484 2018-Nov-19 08:27:32.538747754 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=676E0AAF1B510965EC14B765E786C534115186AADCF0CCD53439DE4531EBC9E0 seq=14479385 2018-Nov-19 08:27:32.538754923 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=676E0AAF1B510965EC14B765E786C534115186AADCF0CCD53439DE4531EBC9E0 seq=14479385 2018-Nov-19 08:27:32.538761727 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=1902B6EE20BC19A14744A01F1AB6ADDAF5E3B95A905EAE9F5C179D6A5D07784B seq=43092484

nbougalis commented 6 years ago

Can you provide us with the output of server_info again please? Thank you.

valanced commented 6 years ago

The same issue:

2018-Nov-22 08:22:04.809591077 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:04.809607126 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:04.809622099 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=7B2E4796D372E18180AE51A6D2930D0165F4EFF556556BAB3F5F0DFB53E3118F seq=43163245
2018-Nov-22 08:22:05.810326381 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=7B2E4796D372E18180AE51A6D2930D0165F4EFF556556BAB3F5F0DFB53E3118F seq=43163245
2018-Nov-22 08:22:05.810388711 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:05.810405641 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:05.810419463 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=7B2E4796D372E18180AE51A6D2930D0165F4EFF556556BAB3F5F0DFB53E3118F seq=43163245
2018-Nov-22 08:22:06.810604238 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=7B2E4796D372E18180AE51A6D2930D0165F4EFF556556BAB3F5F0DFB53E3118F seq=43163245
2018-Nov-22 08:22:06.810671036 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:06.810706052 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:06.810722330 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=7B2E4796D372E18180AE51A6D2930D0165F4EFF556556BAB3F5F0DFB53E3118F seq=43163245
2018-Nov-22 08:22:07.810943196 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=7B2E4796D372E18180AE51A6D2930D0165F4EFF556556BAB3F5F0DFB53E3118F seq=43163245
2018-Nov-22 08:22:07.810979758 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:07.810986095 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:07.810991259 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=7B2E4796D372E18180AE51A6D2930D0165F4EFF556556BAB3F5F0DFB53E3118F seq=43163245
2018-Nov-22 08:22:07.828196035 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=7B2E4796D372E18180AE51A6D2930D0165F4EFF556556BAB3F5F0DFB53E3118F seq=43163245
2018-Nov-22 08:22:07.828235344 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:07.828245946 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:07.828255517 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=7B2E4796D372E18180AE51A6D2930D0165F4EFF556556BAB3F5F0DFB53E3118F seq=43163245
2018-Nov-22 08:22:08.811698858 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=820418E6CE9D2476E3C480D62D49DF453F3A81FE039BC8AE85DF27805BCE01F8 seq=43163246
2018-Nov-22 08:22:08.811772513 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:08.811791804 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:08.811807920 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=820418E6CE9D2476E3C480D62D49DF453F3A81FE039BC8AE85DF27805BCE01F8 seq=43163246
2018-Nov-22 08:22:09.812861914 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=820418E6CE9D2476E3C480D62D49DF453F3A81FE039BC8AE85DF27805BCE01F8 seq=43163246
2018-Nov-22 08:22:09.812908405 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:09.812919982 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3024CC1CEC427917AD7A5B83F155B1C60DD8211E7C41A72D94DCCAD4B6B6F431 seq=43156746
2018-Nov-22 08:22:09.812930222 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=820418E6CE9D2476E3C480D62D49DF453F3A81FE039BC8AE85DF27805BCE01F8 seq=43163246
valanced commented 6 years ago

server_info:

{
    "result": {
        "info": {
            "build_version": "1.1.0",
            "complete_ledgers": "43150480-43151903,43152004-43152021,43152124-43152132,43152234,43152335-43152375,43152477-43152582,43152684-43152861,43152963-43153425,43153527-43154014,43154130-43154142,43154244-43154865,43154967-43155068,43155183-43155379,43155480-43155495,43155558-43162489,43162841-43162923,43163305-43163320",
            "hostid": "node-ripple_container",
            "io_latency_ms": 1,
            "jq_trans_overflow": "81",
            "last_close": {
                "converge_time_s": 2.018,
                "proposers": 23
            },
            "load": {
                "job_types": [
                    {
                        "avg_time": 2,
                        "job_type": "untrustedValidation",
                        "peak_time": 122,
                        "per_second": 3
                    },
                    {
                        "job_type": "ledgerRequest",
                        "per_second": 1
                    },
                    {
                        "job_type": "untrustedProposal",
                        "peak_time": 66,
                        "per_second": 34
                    },
                    {
                        "avg_time": 19,
                        "job_type": "ledgerData",
                        "peak_time": 284,
                        "per_second": 1
                    },
                    {
                        "in_progress": 1,
                        "job_type": "clientCommand",
                        "per_second": 3
                    },
                    {
                        "job_type": "transaction",
                        "peak_time": 9,
                        "per_second": 3
                    },
                    {
                        "job_type": "batch",
                        "peak_time": 9,
                        "per_second": 3
                    },
                    {
                        "avg_time": 11,
                        "in_progress": 1,
                        "job_type": "advanceLedger",
                        "peak_time": 155,
                        "per_second": 2
                    },
                    {
                        "job_type": "fetchTxnData",
                        "per_second": 3
                    },
                    {
                        "avg_time": 4,
                        "job_type": "trustedValidation",
                        "peak_time": 106,
                        "per_second": 4
                    },
                    {
                        "job_type": "writeObjects",
                        "peak_time": 21,
                        "per_second": 28
                    },
                    {
                        "job_type": "trustedProposal",
                        "per_second": 9
                    },
                    {
                        "job_type": "peerCommand",
                        "per_second": 1178
                    },
                    {
                        "job_type": "diskAccess",
                        "peak_time": 5,
                        "per_second": 4
                    },
                    {
                        "job_type": "processTransaction",
                        "per_second": 3
                    },
                    {
                        "job_type": "SyncReadNode",
                        "peak_time": 220,
                        "per_second": 28
                    },
                    {
                        "avg_time": 1,
                        "job_type": "AsyncReadNode",
                        "peak_time": 182,
                        "per_second": 16
                    },
                    {
                        "job_type": "WriteNode",
                        "per_second": 190
                    }
                ],
                "threads": 6
            },
            "load_factor": 1,
            "peer_disconnects": "13",
            "peer_disconnects_resources": "0",
            "peers": 21,
            "pubkey_node": "n9LnomYrhJU2wjCGFq7AAMaAfMbbdTVCJCorjapmaCfjRLqQcYRL",
            "pubkey_validator": "none",
            "published_ledger": 43163320,
            "server_state": "full",
            "state_accounting": {
                "connected": {
                    "duration_us": "25893651193",
                    "transitions": 1
                },
                "disconnected": {
                    "duration_us": "1667962",
                    "transitions": 1
                },
                "full": {
                    "duration_us": "43337254889",
                    "transitions": 414
                },
                "syncing": {
                    "duration_us": "1790888460",
                    "transitions": 415
                },
                "tracking": {
                    "duration_us": "12245301",
                    "transitions": 415
                }
            },
            "time": "2018-Nov-22 08:30:56.313827",
            "uptime": 71039,
            "validated_ledger": {
                "age": 5,
                "base_fee_xrp": 1e-05,
                "hash": "45C31B1D1E0282A8223C6628B407201252FE54575E8DF53B2F1E8C3C8CED07F9",
                "reserve_base_xrp": 20,
                "reserve_inc_xrp": 5,
                "seq": 43163393
            },
            "validation_quorum": 19,
            "validator_list_expires": "2018-Nov-28 00:00:00.000000000"
        },
        "status": "success"
    }
}
busker1004 commented 5 years ago

The same issue: rippled v1.1.1 CentOS 7, 16G ram, 4 cores

Back to normal after restarting ripple service.

2018-Dec-10 06:48:26.969437118 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:26.969450351 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:26.969459780 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=5D1D7537E06EAD4AD5F97BA895290D228BC271AB477C5B3E331A6BE7A6069773 seq=43590571
2018-Dec-10 06:48:27.970152312 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=5D1D7537E06EAD4AD5F97BA895290D228BC271AB477C5B3E331A6BE7A6069773 seq=43590571
2018-Dec-10 06:48:27.970217908 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:27.970235597 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:27.970258767 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=5D1D7537E06EAD4AD5F97BA895290D228BC271AB477C5B3E331A6BE7A6069773 seq=43590571
2018-Dec-10 06:48:28.971041179 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=5D1D7537E06EAD4AD5F97BA895290D228BC271AB477C5B3E331A6BE7A6069773 seq=43590571
2018-Dec-10 06:48:28.971078459 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:28.971088311 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:28.971096750 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=5D1D7537E06EAD4AD5F97BA895290D228BC271AB477C5B3E331A6BE7A6069773 seq=43590571
2018-Dec-10 06:48:29.074972317 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=5D1D7537E06EAD4AD5F97BA895290D228BC271AB477C5B3E331A6BE7A6069773 seq=43590571
2018-Dec-10 06:48:29.075031555 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:29.075045108 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:29.075056385 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=5D1D7537E06EAD4AD5F97BA895290D228BC271AB477C5B3E331A6BE7A6069773 seq=43590571
2018-Dec-10 06:48:29.973018242 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=6DCE305163C798BDC0F2184FBD3F280C90AF5FD2D9046D06AAB2417873726326 seq=43590572
2018-Dec-10 06:48:29.973098767 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:29.973127460 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:29.973164502 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=6DCE305163C798BDC0F2184FBD3F280C90AF5FD2D9046D06AAB2417873726326 seq=43590572
2018-Dec-10 06:48:30.978503359 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=6DCE305163C798BDC0F2184FBD3F280C90AF5FD2D9046D06AAB2417873726326 seq=43590572
2018-Dec-10 06:48:30.978582906 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:30.978596518 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:30.978605183 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=6DCE305163C798BDC0F2184FBD3F280C90AF5FD2D9046D06AAB2417873726326 seq=43590572
2018-Dec-10 06:48:31.979140525 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=6DCE305163C798BDC0F2184FBD3F280C90AF5FD2D9046D06AAB2417873726326 seq=43590572
2018-Dec-10 06:48:31.979182970 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:31.979192928 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:31.979202105 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=6DCE305163C798BDC0F2184FBD3F280C90AF5FD2D9046D06AAB2417873726326 seq=43590572
2018-Dec-10 06:48:32.071597531 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=6DCE305163C798BDC0F2184FBD3F280C90AF5FD2D9046D06AAB2417873726326 seq=43590572
2018-Dec-10 06:48:32.071643146 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:32.071651757 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=83DE70BB7BAF3B24551C53E62423B621881739411D92D1AB75892FFD6F4EBBF7 seq=43588523
2018-Dec-10 06:48:32.071864808 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=6DCE305163C798BDC0F2184FBD3F280C90AF5FD2D9046D06AAB2417873726326 seq=43590572

server_info

{
   "result" : {
      "info" : {
         "build_version" : "1.1.1",
         "complete_ledgers" : "43550986-43590619",
         "hostid" : "localhost.localdomain",
         "io_latency_ms" : 1,
         "jq_trans_overflow" : "0",
         "last_close" : {
            "converge_time_s" : 3.004,
            "proposers" : 25
         },
         "load" : {
            "job_types" : [
               {
                  "avg_time" : 36,
                  "job_type" : "publishAcqLedger",
                  "peak_time" : 174,
                  "per_second" : 1
               },
               {
                  "job_type" : "untrustedValidation",
                  "peak_time" : 3,
                  "per_second" : 4
               },
               {
                  "job_type" : "ledgerRequest",
                  "peak_time" : 8,
                  "per_second" : 7
               },
               {
                  "job_type" : "untrustedProposal",
                  "peak_time" : 5,
                  "per_second" : 26
               },
               {
                  "avg_time" : 4,
                  "in_progress" : 1,
                  "job_type" : "ledgerData",
                  "peak_time" : 101,
                  "per_second" : 23
               },
               {
                  "in_progress" : 1,
                  "job_type" : "clientCommand",
                  "per_second" : 2
               },
               {
                  "job_type" : "transaction",
                  "peak_time" : 2,
                  "per_second" : 5
               },
               {
                  "job_type" : "batch",
                  "per_second" : 5
               },
               {
                  "avg_time" : 2,
                  "in_progress" : 1,
                  "job_type" : "advanceLedger",
                  "peak_time" : 39,
                  "per_second" : 3
               },
               {
                  "job_type" : "fetchTxnData",
                  "peak_time" : 1,
                  "per_second" : 2
               },
               {
                  "job_type" : "trustedValidation",
                  "peak_time" : 3,
                  "per_second" : 6
               },
               {
                  "job_type" : "writeObjects",
                  "peak_time" : 4,
                  "per_second" : 453
               },
               {
                  "job_type" : "trustedProposal",
                  "peak_time" : 2,
                  "per_second" : 8
               },
               {
                  "job_type" : "peerCommand",
                  "peak_time" : 4,
                  "per_second" : 417
               },
               {
                  "job_type" : "diskAccess",
                  "peak_time" : 1,
                  "per_second" : 6
               },
               {
                  "job_type" : "processTransaction",
                  "per_second" : 5
               },
               {
                  "job_type" : "SyncReadNode",
                  "peak_time" : 5,
                  "per_second" : 4980
               },
               {
                  "job_type" : "AsyncReadNode",
                  "peak_time" : 4,
                  "per_second" : 1342
               },
               {
                  "job_type" : "WriteNode",
                  "peak_time" : 1,
                  "per_second" : 1256
               }
            ],
            "threads" : 6
         },
         "load_factor" : 1,
         "peer_disconnects" : "30",
         "peer_disconnects_resources" : "0",
         "peers" : 10,
         "pubkey_node" : "n9LdnDgQPeWFmudcuQZCkpaMeUbWv6rZSpsBKVosNUQpT7L6tUrg",
         "pubkey_validator" : "nHDGyg6ditKheYz3EFLU8SS3hnPkhEGYYj56vK68cithcepD2oVw",
         "published_ledger" : 43590619,
         "server_state" : "proposing",
         "state_accounting" : {
            "connected" : {
               "duration_us" : "141257072",
               "transitions" : 1
            },
            "disconnected" : {
               "duration_us" : "1226700",
               "transitions" : 1
            },
            "full" : {
               "duration_us" : "16048316389",
               "transitions" : 101
            },
            "syncing" : {
               "duration_us" : "782214917",
               "transitions" : 101
            },
            "tracking" : {
               "duration_us" : "264",
               "transitions" : 101
            }
         },
         "time" : "2018-Dec-10 06:51:31.005719",
         "uptime" : 16974,
         "validated_ledger" : {
            "age" : 1,
            "base_fee_xrp" : 1e-05,
            "hash" : "DFFB8E9E0183F5CA276FEAD168828B2721B7B4F74717E5EE31CF4F8A1BD77CC5",
            "reserve_base_xrp" : 20,
            "reserve_inc_xrp" : 5,
            "seq" : 43590620
         },
         "validation_quorum" : 22,
         "validator_list" : {
            "count" : 1,
            "expiration" : "2019-Jan-15 00:00:00.000000000",
            "status" : "active"
         }
      },
      "status" : "success"
   }
}
DaShak commented 5 years ago

I too am plagued with this issue, even with the current rippled build 1.1.2 - Unable to determine hash of ancestor keeps happening after a day or two of operation - currently, my Main Net rippled is non-stop repeating the following:

2018-Dec-17 14:26:35.821735973 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=8C8A368DDEB51620E771B421AA2016048D29E9D217D2F1321348250D138564C8 seq=43724878
2018-Dec-17 14:26:35.821748613 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=A1A6622F8E69EEA8B14483CBCA1388B9D6C5752EF33351A4850ACDF0CAB5684A seq=43723566
2018-Dec-17 14:26:35.821769073 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=57995EF0A00E225109DAD879715D46FCD38CDBF2967FAA4F1AD20B978F8E9938 seq=43765088

Here's the output of server_info:

{
  "result": {
    "info": {
      "build_version": "1.1.2",
      "complete_ledgers": "43695072-43725002,43726212-43726245,43727219-43727229,43727892-43727903,43728301,43728614-43729158,43730312-43730373,43731670-43731679,43731828-43731847,43732215-43732253,43732354-43732413,43732703,43732991-43733426,43734568-43734598,43734799-43734854,43735088-43735119,43735304-43735310,43735518-43735546,43735647-43735685,43735786,43735904-43765116",
      "hostid": "ip-10-50-2-186",
      "io_latency_ms": 1,
      "jq_trans_overflow": "15259",
      "last_close": {
        "converge_time_s": 3.025,
        "proposers": 26
      },
      "load": {
        "job_types": [
          {
            "avg_time": 387,
            "job_type": "untrustedProposal",
            "over_target": true,
            "peak_time": 1608
          },
          {
            "in_progress": 2,
            "job_type": "ledgerData",
            "waiting": 63
          },
          {
            "avg_time": 3,
            "in_progress": 1,
            "job_type": "clientCommand",
            "peak_time": 65,
            "per_second": 1
          },
          {
            "avg_time": 6,
            "job_type": "transaction",
            "peak_time": 33,
            "per_second": 3
          },
          {
            "job_type": "batch",
            "per_second": 2
          },
          {
            "avg_time": 7,
            "job_type": "advanceLedger",
            "peak_time": 202,
            "per_second": 4
          },
          {
            "avg_time": 11,
            "job_type": "fetchTxnData",
            "peak_time": 53,
            "per_second": 1
          },
          {
            "avg_time": 2,
            "job_type": "trustedValidation",
            "peak_time": 28,
            "per_second": 6
          },
          {
            "avg_time": 1,
            "job_type": "writeObjects",
            "peak_time": 35,
            "per_second": 9
          },
          {
            "avg_time": 3,
            "job_type": "trustedProposal",
            "peak_time": 29,
            "per_second": 8
          },
          {
            "job_type": "peerCommand",
            "per_second": 1006
          },
          {
            "avg_time": 1,
            "job_type": "diskAccess",
            "peak_time": 29,
            "per_second": 6
          },
          {
            "job_type": "processTransaction",
            "per_second": 3
          },
          {
            "avg_time": 5,
            "job_type": "SyncReadNode",
            "peak_time": 44,
            "per_second": 34
          },
          {
            "avg_time": 5,
            "job_type": "AsyncReadNode",
            "peak_time": 51,
            "per_second": 137
          },
          {
            "job_type": "WriteNode",
            "per_second": 57
          }
        ],
        "threads": 4
      },
      "load_factor": 14.53515625,
      "load_factor_local": 14.53515625,
      "peer_disconnects": "127",
      "peer_disconnects_resources": "0",
      "peers": 21,
      "pubkey_node": "n9KPtAo9w1S1Ac9JTMbUw1pCCR4fBkymi6cCpS6WpK9ey46zejRp",
      "pubkey_validator": "none",
      "server_state": "full",
      "state_accounting": {
        "connected": {
          "duration_us": "253511905",
          "transitions": 1
        },
        "disconnected": {
          "duration_us": "1333023",
          "transitions": 1
        },
        "full": {
          "duration_us": "253086516628",
          "transitions": 10
        },
        "syncing": {
          "duration_us": "51802412",
          "transitions": 10
        },
        "tracking": {
          "duration_us": "11",
          "transitions": 10
        }
      },
      "time": "2018-Dec-17 14:28:11.624120",
      "uptime": 253393,
      "validated_ledger": {
        "age": 1,
        "base_fee_xrp": 1e-05,
        "hash": "A14A4D9711CFBE7DD59EC13C02C870073C3FA876EEC638C9CCEF89B3A14EB9C3",
        "reserve_base_xrp": 20,
        "reserve_inc_xrp": 5,
        "seq": 43765116
      },
      "validation_quorum": 21,
      "validator_list": {
        "count": 1,
        "expiration": "2019-Jan-15 00:00:00.000000000",
        "status": "active"
      }
    },
    "status": "success"
  }
}

rippled did not stop after issuing the command via systemctl, so I tried sent stop via the admin method to the port_ws_admin_local per https://developers.ripple.com/stop.html but the response was Service Unavailable. However, rippled appears to have stopped on its own after a few minutes, while I was writing this Comment…

The AWS Monitoring for the Volume looks suspect to me - it is a gp2 SSD (60 GiB, 180 IOPS) - I can share if it helps…

My Test Net rippled with nearly identical cfg runs continuously without this issue; using RocksDB.

I can share my cfg or anything that helps; please advise…

CC: @nbougalis

qiluge commented 5 years ago

I have also got the Service Unavailable problem. After ripple sync node run some hours, the server has no response to rpc request, and after some time, the server response is normal... Reference to https://github.com/ripple/rippled/issues/2900

Merlz commented 5 years ago

We are also seeing this happening now, on all 6 of our servers (5 partial history and 1 full history), it started recently but was running fine for months and nothing changed in the configuration. Even with the db removed from one of the partial history servers (750000 ledgers) and syncing a clean db, it started showing Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash= in the logs almost immediately after starting to sync with the network. Disks on the partial history servers are 750 GB, 6000 IOPS and they are not using close to the full IOPS, the full history server has a 13 TB disk with 16000 IOPS. All servers have been rebooted (terminated/recreated) with no difference.

This started happening on version 1.2.2 and even on 1.2.4 it continues. Any ideas why this is happening?

@nbougalis

gituser commented 5 years ago

I can confirm I have the same experience as @Merlz, quite annoying rippled from time to time just stuck and stops working on NVME drives.

Happens on latest v1.2.4 version with default settings. I don't keep all history only 2000 last ledgers.

qiluge commented 5 years ago

Please use more system resource for rippled, because after I changed my mechanical hard disk to SSD and increased the number of CPUs, this problem was solved.

gituser commented 5 years ago

What do you mean more?

I have already NVME drives there and i9-9900K cpus.

It's not a solution btw as it worked before just fine.

qiluge commented 5 years ago

What do you mean more?

I have already NVME drives there and i9-9900K cpus.

It's not a solution btw as it worked before just fine.

OK，and this is just my experience. I'm sorry if that can't help you.

codemandan commented 5 years ago

I've just started a new validator and seemingly ran into this issue after a day. Validator list also shows me as being on a chain not main now. Not sure if it's related to above or misconfigured, it was running fine for some hours after initial setup.

(a reboot of rippled kicked me back onto main and stopped mass spamming the wrn message)

Portion of log:

2019-Jul-18 00:58:12.848719957 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=CBF3C264D80CCC49C2085809DB7485530BF429EC5607BAE6B8BB62EF90EBD087 seq=48717549
2019-Jul-18 00:58:12.848747633 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=3CDB4B2427E659CFBC1C07823D4E4DDCD85658EC64E905D028397D98E90E6EBB seq=48725635
2019-Jul-18 00:58:12.848770724 Validations:WRN Unable to determine hash of ancestor seq=1 from ledger hash=CBF3C264D80CCC49C2085809DB7485530BF429EC5607BAE6B8BB62EF90EBD087 seq=48717549

show_info

"result" : {
      "info" : {
         "build_version" : "1.2.4",
         "complete_ledgers" : "48723369-48726174",
         "hostid" : "4f070582ce13",
         "io_latency_ms" : 1,
         "jq_trans_overflow" : "185",
         "last_close" : {
            "converge_time_s" : 2.009,
            "proposers" : 0
         },
         "load" : {
            "job_types" : [
               {
                  "job_type" : "ledgerRequest",
                  "per_second" : 5
               },
               {
                  "job_type" : "untrustedProposal",
                  "per_second" : 36
               },
               {
                  "avg_time" : 1,
                  "job_type" : "ledgerData",
                  "peak_time" : 20,
                  "per_second" : 4
               },
               {
                  "in_progress" : 1,
                  "job_type" : "clientCommand"
               },
               {
                  "job_type" : "transaction",
                  "per_second" : 12
               },
               {
                  "job_type" : "batch",
                  "per_second" : 11
               },
               {
                  "job_type" : "advanceLedger",
                  "peak_time" : 76,
                  "per_second" : 11
               },
               {
                  "job_type" : "trustedValidation",
                  "peak_time" : 3,
                  "per_second" : 4
               },
               {
                  "job_type" : "writeObjects",
                  "peak_time" : 3,
                  "per_second" : 75
               },
               {
                  "job_type" : "trustedProposal",
                  "per_second" : 11
               },
               {
                  "job_type" : "peerCommand",
                  "per_second" : 1140
               },
               {
                  "job_type" : "diskAccess",
                  "per_second" : 4
               },
               {
                  "job_type" : "processTransaction",
                  "per_second" : 12
               },
               {
                  "job_type" : "SyncReadNode",
                  "per_second" : 99
               },
               {
                  "job_type" : "AsyncReadNode",
                  "per_second" : 48
               },
               {
                  "job_type" : "WriteNode",
                  "per_second" : 158
               }
            ],
            "threads" : 6
         },
         "load_factor" : 1,
         "peer_disconnects" : "122",
         "peer_disconnects_resources" : "2",
         "peers" : 21,
         "pubkey_node" : "n9KWDbiDSmffJqGdbFS3ZSSggtTesuRjktJTchzetrtq1mUhxkAX",
         "pubkey_validator" : "nHUA6xFgTKHhK3fp6HAqehsA3qAZ3kWQUincPGi7QDzAzXmHuFJF",
         "server_state" : "proposing",
         "server_state_duration_us" : "42098122478",
         "state_accounting" : {
            "connected" : {
               "duration_us" : "71029937",
               "transitions" : 1
            },
            "disconnected" : {
               "duration_us" : "3380681",
               "transitions" : 1
            },
            "full" : {
               "duration_us" : "66932488272",
               "transitions" : 8
            },
            "syncing" : {
               "duration_us" : "14934985",
               "transitions" : 7
            },
            "tracking" : {
               "duration_us" : "1",
               "transitions" : 7
            }
         },
         "time" : "2019-Jul-18 01:16:22.609435",
         "uptime" : 67021,
         "validated_ledger" : {
            "age" : 2,
            "base_fee_xrp" : 1e-05,
            "hash" : "EF5FBA5E1F04D1DA49E833B1F1531AD35B248CDA37FB2853FA9C7CB019DAB174",
            "reserve_base_xrp" : 20,
            "reserve_inc_xrp" : 5,
            "seq" : 48726174
         },
         "validation_quorum" : 26,
         "validator_list" : {
            "count" : 1,
            "expiration" : "2019-Aug-14 00:00:00.000000000",
            "status" : "active"
         }
      },
      "status" : "success"
   }
nbougalis commented 4 years ago

I believe that this issue has been corrected. It's not unrealistic for this message to pop up periodically. I am going to wait to see if anyone is still experiencing issues, and will close a month from now.