search

dashpay / dash

Dash - Reinventing Cryptocurrency
https://www.dash.org
MIT License
1.49k stars 1.2k forks source link

Network froze on 1874879 May 22, 2023 4:02:30 AM UTC / daemon can't start too #5383

Open gades opened 1 year ago

gades commented 1 year ago

Daemon can't start in case restart - invalid blocks and after resync it stops on 1874879

knst commented 1 year ago

There was a problem during v19 activation of Dash Core. The chain is stalled and currently not producing blocks. We have everyone investigating the issue.

https://twitter.com/SamWestrich/status/1660513873546596354

gades commented 1 year ago

"v19": { "status": "active", "start_time": 1682380800, "timeout": 1714003200, "since": 1874880 }

yeah, this issue is in bip9 softforks and  why was not tested it in testnet before roll out in main net?
gituser commented 1 year ago

After update to v19.0.0 from v18.2.2 getting this error:

2023-05-22T10:41:54Z Using obfuscation key for /home/dash/.dash/chainstate: 19f6def09d7d0cc9
2023-05-22T10:41:55Z Loaded best chain: hashBestChain=000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7 height=1874880 date=2023-05-22T04:13:10Z progress=0.999918
2023-05-22T10:41:55Z CDeterministicMNManager::MigrateDBIfNeeded -- upgrading DB to migrate MN type
2023-05-22T10:41:55Z CDeterministicMNManager::MigrateDBIfNeeded -- migration already done. skipping.
2023-05-22T10:41:55Z init message: Verifying blocks...
2023-05-22T10:41:55Z Verifying last 6 blocks at level 3
2023-05-22T10:41:55Z [0%]...[16%]...[33%]...ERROR: UndoSpecialTxsInBlock -- failed: UpdateMN: Can't update a masternode e01979c6ce16c95541310d879f91e2324164bab362f5d21f903f33941fa0b5e5 with a duplicate pubKeyOperator=000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

2023-05-22T10:42:01Z ERROR: VerifyDB(): *** irrecoverable inconsistency in block data at 1874878, hash=0000000000000009be0f8add92c7d1d92654806f76f983f1e089a20cc0597926
2023-05-22T10:42:01Z : Corrupted block database detected.
Please restart with -reindex or -reindex-chainstate to recover.
2023-05-22T10:42:01Z Aborted block database rebuild. Exiting.
2023-05-22T10:42:01Z PrepareShutdown: In progress...
2023-05-22T10:42:01Z cl-schdlr thread exit
2023-05-22T10:42:01Z scheduler thread exit
gades commented 1 year ago

After update to v19.0.0 from v18.2.2 getting this error:

2023-05-22T10:41:54Z Using obfuscation key for /home/dash/.dash/chainstate: 19f6def09d7d0cc9
2023-05-22T10:41:55Z Loaded best chain: hashBestChain=000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7 height=1874880 date=2023-05-22T04:13:10Z progress=0.999918
2023-05-22T10:41:55Z CDeterministicMNManager::MigrateDBIfNeeded -- upgrading DB to migrate MN type
2023-05-22T10:41:55Z CDeterministicMNManager::MigrateDBIfNeeded -- migration already done. skipping.
2023-05-22T10:41:55Z init message: Verifying blocks...
2023-05-22T10:41:55Z Verifying last 6 blocks at level 3
2023-05-22T10:41:55Z [0%]...[16%]...[33%]...ERROR: UndoSpecialTxsInBlock -- failed: UpdateMN: Can't update a masternode e01979c6ce16c95541310d879f91e2324164bab362f5d21f903f33941fa0b5e5 with a duplicate pubKeyOperator=000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

2023-05-22T10:42:01Z ERROR: VerifyDB(): *** irrecoverable inconsistency in block data at 1874878, hash=0000000000000009be0f8add92c7d1d92654806f76f983f1e089a20cc0597926
2023-05-22T10:42:01Z : Corrupted block database detected.
Please restart with -reindex or -reindex-chainstate to recover.
2023-05-22T10:42:01Z Aborted block database rebuild. Exiting.
2023-05-22T10:42:01Z PrepareShutdown: In progress...
2023-05-22T10:42:01Z cl-schdlr thread exit
2023-05-22T10:42:01Z scheduler thread exit

the same issue with v19 and you can fix by downloading blocks from network and it will work until daemon will up

gituser commented 1 year ago

the same issue with v19 and you can fix by downloading blocks from network and it will work until daemon will up

is there any way to fix this without resync?

qwizzie commented 1 year ago

the same issue with v19 and you can fix by downloading blocks from network and it will work until daemon will up

is there any way to fix this without resync?

Untill it becomes clear what exactly the issue is and how they plan to address the issue (hotfix ?) we should just wait. Dash blockchain explorer (https://chainz.cryptoid.info/dash/) is also still stuck (indicating the blockchain is still stallled), so i don't think a resync will help much at this moment.

gades commented 1 year ago

They did patch https://github.com/dashpay/dash/commit/ea184524ac1c93387217413dc37cbf7739d7da3e and new version is accepting blocks from miner who is using last one ...

Har01d commented 1 year ago

Upgrading to 19.1.0 doesn't help, the node is stuck at height=1874880 invalidateblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb and reconsiderblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb don't help either.

qwizzie commented 1 year ago

Upgrading to 19.1.0 doesn't help, the node is stuck at height=1874880 invalidateblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb and reconsiderblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb don't help either.

Yeah, i ended up giving my masternodes a reindex command after updating to v19.1 Spork 23 (Quorum PoSe) is disabled, so no worries about getting PoSe scored during reindex with regards to masternodes.

PastaPastaPasta commented 1 year ago

current VALID height and hash I see is / are { "height": 1874884, "hash": "000000000000005c3c3370b96c2a4f58e7062a050be45479dd58cd3a91121950", "difficulty": 32942678.88782359, "chainwork": "0000000000000000000000000000000000000000000084f03183f1747f340482", "branchlen": 0, "forkpoint": "000000000000005c3c3370b96c2a4f58e7062a050be45479dd58cd3a91121950", "status": "active" }, { "height": 1874880, "hash": "000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7", "difficulty": 117437257.0405482, "chainwork": "0000000000000000000000000000000000000000000084f02826d7a9be295cf4", "branchlen": 1, "forkpoint": "0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb", "status": "valid-fork" }

qwizzie commented 1 year ago

I am seeing a lot (and i mean a lot) of 'CActiveMasternodeManager::Init -- ERROR: Local address does not match the address from ProTx' messages in my debug.log after i updated to v19.1 and issued a reindex. I checked a few masternodes of mine and those i checked all had that error message in their debug.log

Update : decided to just remove the blockchain and most of the .dat files and download and sync from scratch.

Har01d commented 1 year ago

Upgrading to 19.1.0 doesn't help, the node is stuck at height=1874880 invalidateblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb and reconsiderblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb don't help either.

It suddenly started to sync and it seems to be working fine now.

maayank commented 1 year ago

With 19.1. Still froze for us. Tried to invalidate and reconsider (even using earlier blocks) and didn't help. Getblockcount, getpeerinfo and getblockchaininfo after the break. Would appreciate all help

root@dash:/# dash-cli -datadir=/data/dash getblockcount
1874880
root@dash:/# dash-cli -datadir=/data/dash getpeerinfo  
[
  {
    "id": 1,
    "addr": "82.211.25.172:9999",
    "addrlocal": "3.21.200.4:39512",
    "addrbind": "172.18.0.3:39512",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799294,
    "lastrecv": 1684799339,
    "bytessent": 1488,
    "bytesrecv": 18787,
    "conntime": 1684799287,
    "timeoffset": 0,
    "pingtime": 0.229181,
    "minping": 0.229181,
    "version": 70227,
    "subver": "/Dash Core:19.0.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874879,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getheaders2": 1085,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 15645,
      "dsq": 166,
      "headers2": 25,
      "inv": 244,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 2,
    "addr": "51.91.75.25:9999",
    "addrlocal": "3.21.200.4:47566",
    "addrbind": "172.18.0.3:47566",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799294,
    "lastrecv": 1684799357,
    "bytessent": 403,
    "bytesrecv": 20077,
    "conntime": 1684799289,
    "timeoffset": 0,
    "pingtime": 0.121307,
    "minping": 0.121307,
    "version": 70227,
    "subver": "/Dash Core:19.0.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874879,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 17328,
      "inv": 194,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 3,
    "addr": "68.183.188.215:9999",
    "addrlocal": "3.21.200.4:38284",
    "addrbind": "172.18.0.3:38284",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799306,
    "lastrecv": 1684799346,
    "bytessent": 403,
    "bytesrecv": 20528,
    "conntime": 1684799301,
    "timeoffset": -1,
    "pingtime": 0.47413,
    "minping": 0.47413,
    "version": 70227,
    "subver": "/Dash Core:19.1.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874990,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 16387,
      "dsq": 166,
      "getheaders2": 1085,
      "inv": 183,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 4,
    "addr": "139.162.105.155:9999",
    "addrlocal": "3.21.200.4:52856",
    "addrbind": "172.18.0.3:52856",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799306,
    "lastrecv": 1684799352,
    "bytessent": 403,
    "bytesrecv": 19590,
    "conntime": 1684799302,
    "timeoffset": 0,
    "pingtime": 0.300366,
    "minping": 0.300366,
    "version": 70223,
    "subver": "/Dash Core:18.0.1/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874990,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 15597,
      "dsq": 170,
      "getheaders2": 1085,
      "inv": 183,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 5,
    "addr": "95.183.51.141:9999",
    "addrlocal": "3.21.200.4:48508",
    "addrbind": "172.18.0.3:48508",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799353,
    "lastrecv": 1684799346,
    "bytessent": 443,
    "bytesrecv": 18533,
    "conntime": 1684799303,
    "timeoffset": -1,
    "pingtime": 0.124047,
    "minping": 0.124047,
    "version": 70227,
    "subver": "/Dash Core:19.0.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874879,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "addrv2": 40,
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 15538,
      "dsq": 166,
      "inv": 122,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 6,
    "addr": "195.181.211.64:9999",
    "addrbind": "172.18.0.3:34134",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799306,
    "lastrecv": 1684799339,
    "bytessent": 403,
    "bytesrecv": 18367,
    "conntime": 1684799303,
    "timeoffset": 0,
    "pingtime": 0.205568,
    "minping": 0.205568,
    "version": 70227,
    "subver": "/Dash Core:19.0.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874879,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 15538,
      "inv": 122,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 8,
    "addr": "150.136.103.179:9999",
    "addrlocal": "3.21.200.4:52084",
    "addrbind": "172.18.0.3:52084",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799343,
    "lastrecv": 1684799339,
    "bytessent": 443,
    "bytesrecv": 17876,
    "conntime": 1684799305,
    "timeoffset": 0,
    "pingtime": 0.023654,
    "minping": 0.023654,
    "version": 70227,
    "subver": "/Dash Core:19.1.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1875067,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "addrv2": 40,
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 15199,
      "inv": 122,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 10,
    "addr": "88.99.11.5:9999",
    "addrlocal": "3.21.200.4:38156",
    "addrbind": "172.18.0.3:38156",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": true,
    "lastsend": 1684799312,
    "lastrecv": 1684799346,
    "bytessent": 403,
    "bytesrecv": 21154,
    "conntime": 1684799307,
    "timeoffset": 0,
    "pingtime": 0.200404,
    "minping": 0.200404,
    "version": 70227,
    "subver": "/Dash Core:19.1.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874990,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getaddr": 24,
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 17013,
      "dsq": 166,
      "getheaders2": 1085,
      "inv": 183,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 11,
    "addr": "135.181.52.145:9999",
    "addrlocal": "3.21.200.4:38928",
    "addrbind": "172.18.0.3:38928",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": false,
    "lastsend": 1684799312,
    "lastrecv": 1684799334,
    "bytessent": 354,
    "bytesrecv": 3933,
    "conntime": 1684799308,
    "timeoffset": 0,
    "pingtime": 0.237356,
    "minping": 0.237356,
    "version": 70227,
    "subver": "/Dash Core:19.1.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874990,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 80,
      "getheaders2": 1085,
      "inv": 61,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  },
  {
    "id": 13,
    "addr": "188.40.163.23:9999",
    "addrlocal": "3.21.200.4:42154",
    "addrbind": "172.18.0.3:42154",
    "services": "0000000000000c05",
    "servicesnames": [
      "NETWORK",
      "BLOOM",
      "NETWORK_LIMITED",
      "HEADERS_COMPRESSED"
    ],
    "relaytxes": false,
    "lastsend": 1684799318,
    "lastrecv": 1684799334,
    "bytessent": 354,
    "bytesrecv": 3893,
    "conntime": 1684799313,
    "timeoffset": 0,
    "pingtime": 0.200535,
    "minping": 0.200535,
    "version": 70227,
    "subver": "/Dash Core:19.1.0/",
    "inbound": false,
    "addnode": false,
    "masternode": false,
    "startingheight": 1874990,
    "banscore": 0,
    "synced_headers": -1,
    "synced_blocks": -1,
    "inflight": [
    ],
    "whitelisted": false,
    "permissions": [
    ],
    "bytessent_per_msg": {
      "getsporks": 24,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "sendheaders2": 24,
      "verack": 24,
      "version": 161
    },
    "bytesrecv_per_msg": {
      "addrv2": 40,
      "getheaders2": 1085,
      "inv": 61,
      "mnauth": 152,
      "ping": 32,
      "pong": 32,
      "sendaddrv2": 24,
      "sendcmpct": 33,
      "senddsq": 25,
      "sendheaders2": 24,
      "spork": 2200,
      "verack": 24,
      "version": 161
    }
  }
]
root@dash:/# dash-cli -datadir=/data/dash getblockchaininfo
{
  "chain": "main",
  "blocks": 1874880,
  "headers": 1874880,
  "bestblockhash": "000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7",
  "difficulty": 117437257.0405482,
  "mediantime": 1684727247,
  "verificationprogress": 0.9997502737722491,
  "initialblockdownload": true,
  "chainwork": "0000000000000000000000000000000000000000000084f02826d7a9be295cf4",
  "size_on_disk": 34076801844,
  "pruned": false,
  "softforks": [
    {
      "id": "bip34",
      "version": 2,
      "reject": {
        "status": true
      }
    },
    {
      "id": "bip66",
      "version": 3,
      "reject": {
        "status": true
      }
    },
    {
      "id": "bip65",
      "version": 4,
      "reject": {
        "status": true
      }
    }
  ],
  "bip9_softforks": {
    "csv": {
      "status": "active",
      "start_time": 1486252800,
      "timeout": 1517788800,
      "since": 622944
    },
    "dip0001": {
      "status": "active",
      "start_time": 1508025600,
      "timeout": 1539561600,
      "since": 782208
    },
    "bip147": {
      "status": "active",
      "start_time": 1524477600,
      "timeout": 1556013600,
      "since": 939456
    },
    "dip0003": {
      "status": "active",
      "start_time": 1546300800,
      "timeout": 1577836800,
      "since": 1028160
    },
    "dip0008": {
      "status": "active",
      "start_time": 1557878400,
      "timeout": 1589500800,
      "since": 1088640
    },
    "realloc": {
      "status": "active",
      "start_time": 1601510400,
      "timeout": 1633046400,
      "since": 1374912
    },
    "dip0020": {
      "status": "active",
      "start_time": 1625097600,
      "timeout": 1656633600,
      "since": 1516032
    },
    "dip0024": {
      "status": "active",
      "start_time": 1660521600,
      "timeout": 1692057600,
      "since": 1737792
    },
    "v19": {
      "status": "defined",
      "start_time": 1686700800,
      "timeout": 1718323200,
      "since": 0
    }
  },
  "warnings": ""
}
UdjinM6 commented 1 year ago

@maayank pls try disconnecting a couple of nodes so that the first node in getpeerinfo list is Dash Core 19.1.0 (e.g. do disconnectnode "" 1 and disconnectnode "" 2) and see if helps.

PastaPastaPasta commented 1 year ago

If you are stuck on block 1874880, you can try to perform the following

grab this text here: https://gist.github.com/PastaPastaPasta/ace08090e6034b3cd1b7f1379776be7f

and run submitblock

joslee7410 commented 1 year ago

Anyone have other solution to share?

I'm still stucking at Block 1874879 Hash 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb

Please help, thank you.

starwill commented 1 year ago

Upgrading to 19.1.0 doesn't help, the node is stuck at height=1874880 invalidateblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb and reconsiderblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb don't help either.

It suddenly started to sync and it seems to be working fine now.

how long have you waited until it work normally ? I have upgrade to v19.1.0 for 2 hours but it's still stucking at 1874879. have you take any other action to recover it？

gades commented 1 year ago

how long have you waited until it work normally ? I have upgrade to v19.1.0 for 2 hours but it's still stucking at 1874879. have you take any other action to recover it？

I did following steps: 1) stop node 2) update it to v19.1.0 3) remove all files in ~/.dashcore/ except dash.conf / wallet.dat 4) start node

other steps like invalidateblock / reconsiderblock / submitblock / disconnectnode were unsuccessful

gkardava commented 1 year ago

dash v19.1.0 crashed with following error.

2023-05-23T08:12:45Z [default wallet] Submitting wtx ****************************** to mempool for relay
2023-05-23T08:12:46Z [ProcessBlock] h[1874880] numCommitmentsRequired[0] numCommitmentsInNewBlock[0]
2023-05-23T08:12:47Z CDeterministicMNManager::ProcessBlock -- Wrote snapshot. nHeight=1874880, mapCurMNs.allMNsCount=4400
2023-05-23T08:12:47Z ERROR: ConnectBlock(DASH): ProcessSpecialTxsInBlock for block 000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7 failed with bad-cbtx-mnmerkleroot (code 16)
2023-05-23T08:12:47Z InvalidChainFound: invalid block=000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7  height=1874880  log2_work=79.05461097  date=2023-05-22T04:13:10Z
2023-05-23T08:12:47Z InvalidChainFound:  current best=0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb  height=1874879  log2_work=79.05460981  date=2023-05-22T04:02:30Z
2023-05-23T08:12:47Z ERROR: ConnectTip: ConnectBlock 000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7 failed, bad-cbtx-mnmerkleroot (code 16)
2023-05-23T08:12:47Z InvalidChainFound: invalid block=000000000000001e28817f2896e1dcf993dfab73f4b80a64e9e3be5680d2b3d7  height=1874880  log2_work=79.05461097  date=2023-05-22T04:13:10Z
2023-05-23T08:12:47Z InvalidChainFound:  current best=0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb  height=1874879  log2_work=79.05460981  date=2023-05-22T04:02:30Z
2023-05-23T08:12:47Z Filling coin cache with masternode UTXOs...
2023-05-23T08:12:47Z Filling coin cache with masternode UTXOs: done in 267ms
2023-05-23T08:12:47Z [default wallet] AddToWallet ******************************
2023-05-23T08:12:49Z [default wallet] AddToWallet ******************************
2023-05-23T08:12:49Z Imported mempool transactions from disk: 342 succeeded, 1 failed, 0 expired, 1 already there, 0 waiting for initial broadcast
2023-05-23T08:12:49Z loadblk thread exit
2023-05-23T08:12:52Z New outbound peer connected: version: 70227, blocks=1875183, peer=0 (full-relay)
2023-05-23T08:12:52Z Synchronizing blockheaders, height: 1875183 (~100.00%)
2023-05-23T08:12:53Z [ProcessBlock] h[1874880] numCommitmentsRequired[0] numCommitmentsInNewBlock[0]
2023-05-23T08:12:53Z CDeterministicMNManager::ProcessBlock -- Wrote snapshot. nHeight=1874880, mapCurMNs.allMNsCount=4400
2023-05-23T08:12:53Z Posix Signal: Segmentation fault
No debug information available for stacktrace. You should add debug information and then run:
gades commented 1 year ago

dash v19.1.0 crashed with following error.

can you try to remove all files in ~/.dashcore/ except dash.conf / wallet.dat and start again?

starwill commented 1 year ago

how long have you waited until it work normally ? I have upgrade to v19.1.0 for 2 hours but it's still stucking at 1874879. have you take any other action to recover it？

I did following steps:

  1. stop node
  2. update it to v19.1.0
  3. remove all files in ~/.dashcore/ except dash.conf / wallet.dat
  4. start node

other steps like invalidateblock / reconsiderblock / submitblock / disconnectnode were unsuccessful

did you mean that you clear the chaindata and resync the chain?

gades commented 1 year ago

did you mean that you clear the chaindata and resync the chain?

yes, it's only one way which solve this issue for me

maayank commented 1 year ago

Just wanted to say that by now our node resumed syncing

Har01d commented 1 year ago

Upgrading to 19.1.0 doesn't help, the node is stuck at height=1874880 invalidateblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb and reconsiderblock 0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb don't help either.

It suddenly started to sync and it seems to be working fine now.

how long have you waited until it work normally ? I have upgrade to v19.1.0 for 2 hours but it's still stucking at 1874879. have you take any other action to recover it？

Several hours. We also tried to delete banlist.dat and peers.dat but it didn't seem to help.

gades commented 1 year ago

Several hours. We also tried to delete banlist.dat and peers.dat but it didn't seem to help.

Resyncing node from scratch was approx 2-3 hours and our nodes are up. But strange , before we start do it the count blocks were higher than in the main net

@PastaPastaPasta will there new update because I didn't saw any fixes for v19 and only one change is reschedule start voting to 15 June ?

Crypto2 commented 1 year ago

You should only have to launch with -reindex, not do a full resync.

PastaPastaPasta commented 1 year ago

Yes, there will be an 19.2 once we have proper fixes for the issues we saw.

impishian commented 1 year ago

It's been at least 5 days, still not fixed❓

Is the dash chain dead❓

No one uses this chain anymore❓ 😴

PastaPastaPasta commented 1 year ago

... No, the actual issues are being fixed in PRs such as https://github.com/dashpay/dash/pull/5397, https://github.com/dashpay/dash/pull/5395, https://github.com/dashpay/dash/pull/5392 & https://github.com/dashpay/dash/pull/5386

However, the network is, and has been since v19.1 was released a week ago, completely operational.

Let's remember to keep the GitHub professional and respectful.

impishian commented 1 year ago

$ dash-cli --version Dash Core RPC client version v19.1.0

$ dash-cli getblockchaininfo|head { "chain": "main", "blocks": 1874879, "headers": 1874879, "bestblockhash": "0000000000000013cdc708723f111c6d34effd9ea663e62eaaa5c9ff299800cb", "difficulty": 119636794.055944, "mediantime": 1684727208, "verificationprogress": 0.9975710346798932, "initialblockdownload": true, "chainwork": "0000000000000000000000000000000000000000000084f02126dd60b97ef9c4",

remove all files in ~/.dashcore/ except dash.conf / wallet.dat

resync, three times, same result.

Crypto2 commented 1 year ago

Make sure you reindex (-reindex or reindex=1 in your config file), then it should resume fine. You make have to do a reconsiderblock on the correct block too.

impishian commented 1 year ago

before "remove all files ... and resync", already tried reindex, invalidateblock/reconsiderblock, but it doesn't work.

qwizzie commented 1 year ago

In two days a new Dash bootstrap file will be created (June 1 00:00:01 UTC 2023) Maybe that specific June bootstrap file can help people, who are currently still struggling.

See : https://github.com/UdjinM6/dash-bootstrap

It does mean waiting for two more days.

With me the removal of all files in ~/.dashcore/ (except dash.conf / .lock / settings.json / sentinel folder) did the trick for my masternodes. If people are not running masternodes, the removal of all files in ~/.dashcore/ (except dash.conf / wallet.dat / .lock / settings.json) should do the trick.

petebus commented 1 year ago

I tried run invalidateblock and reconsiderblock and then i -reindex my wallet and syncrhonize was succesfully, but i get this warning:

Warning: unknown new rules activated (versionbit 8)

What does it mean? Is there cause of concern?

Dash Core 19.1.0

knst commented 1 year ago

Warning: unknown new rules activated (versionbit 8)

That's an artefact of postponing hard fork v19 (bit 8). This warning would disapper in a future 19.2 release.

petebus commented 1 year ago

That's an artefact of postponing hard fork v19 (bit 8). This warning would disapper in a future 19.2 release.

Just in case, I've updated Dash Core to 19.2 and i still see this warning.

chostorg commented 1 year ago

Can you guys help BTCPAYSERVER bump their version to latest? I'm stuck with this bug. According to https://github.com/btcpayserver/btcpayserver-docker/issues/803 you should help them somehow.

thephez commented 1 year ago

Can you guys help BTCPAYSERVER bump their version to latest? I'm stuck with this bug. According to btcpayserver/btcpayserver-docker#803 you should help them somehow.

There is an open PR there to do this. See https://github.com/btcpayserver/btcpayserver-docker/pull/800

impishian commented 1 year ago

Block sync stopped at 1899071.

reconsiderblock 0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae remove peers.dat

both, not helped!

$ dash-cli --version Dash Core RPC client version v19.2.0

$ dash-cli getblockchaininfo|head { "chain": "main", "blocks": 1899071, "headers": 1899071, "bestblockhash": "0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6", "difficulty": 83302114.74555434, "mediantime": 1688597376, "verificationprogress": 0.9999679197411995, "initialblockdownload": false, "chainwork": "00000000000000000000000000000000000000000000877f1fb6fb18c7e7fb79",

$ dash-cli getblockhash 1899071 0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6

1899071 hash： https://insight.dash.org/insight/block/0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6

1899072 hash： https://insight.dash.org/insight/block/0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae

$ grep "is marked invalid" debug.log ... ... ... 0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae is marked invalid

Scumwong commented 1 year ago

Block sync stopped at 1899071.

reconsiderblock 0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae remove peers.dat

both, not helped!

$ dash-cli --version Dash Core RPC client version v19.2.0

$ dash-cli getblockchaininfo|head { "chain": "main", "blocks": 1899071, "headers": 1899071, "bestblockhash": "0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6", "difficulty": 83302114.74555434, "mediantime": 1688597376, "verificationprogress": 0.9999679197411995, "initialblockdownload": false, "chainwork": "00000000000000000000000000000000000000000000877f1fb6fb18c7e7fb79",

$ dash-cli getblockhash 1899071 0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6

1899071 hash： https://insight.dash.org/insight/block/0000000000000022cd5c8d422c2e9b3a876dee260402e8850d95ca3bf64cb9b6

1899072 hash： https://insight.dash.org/insight/block/0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae

$ grep "is marked invalid" debug.log ... ... ... 0000000000000015e32e73052d663626327004c81c5c22cb8b42c361015c0eae is marked invalid

the same problem here . have you resloved it?

riongull commented 1 year ago

You need to update to v19.2 if you haven't already and then reindex your chain:

  1. Add reindex=1 to dash.conf
  2. Restart dashd and let it reindex and sync (can take several hours)
  3. Stop dashd and remove or comment out reindex=1
  4. Restart dashd
Scumwong commented 1 year ago

You need to reindex your chain.

  1. Add reindex=1 to dash.conf
  2. Restart dashd and let it reindex and sync (can take several hours)
  3. Remove or comment out reindex=1
  4. Restart dashd

It works,thank you very much