Masternodes lose sync

[KenSharp]

After a number of weeks running masternodes without issue, they lose sync and refuse to continue. I have two MN running and they both lose sync in the same place and hence at the same time.

The only option ever offered is to nuke the chainstate and re-sync, which is not a viable option and should not need to be done.

Attempting to invalidate the first invalid block would seem to be the logical solution, but the MN never continues and the result is nothing useful.

 {
    "height": 377340,
    "hash": "00000000000022d2b554dd6e394e1cce6cecf2f7918f3deb2875a23f8352c0c9",
    "difficulty": 242509.9156163001,
    "chainwork": "000000000000000000000000000000000000000000000080a65c1b4fb5cdfae4",
    "branchlen": 3,
    "forkpoint": "0000000000001378e25f8f992865d95aadd1b6a3cfe93889f8bee416c5c248f4",
    "status": "conflicting"
  },
  {
    "height": 377338,
    "hash": "00000000000004d68329929c610d918efe5725d42f8c4859d624924d2c500f6f",
    "difficulty": 267678.7096827597,
    "chainwork": "000000000000000000000000000000000000000000000080a6548436f42afee7",
    "branchlen": 1,
    "forkpoint": "0000000000001378e25f8f992865d95aadd1b6a3cfe93889f8bee416c5c248f4",
    "status": "conflicting"
  },
  {
    "height": 377337,
    "hash": "0000000000001378e25f8f992865d95aadd1b6a3cfe93889f8bee416c5c248f4",
    "difficulty": 280082.1197386601,
    "chainwork": "000000000000000000000000000000000000000000000080a6506e9428da6ea4",
    "branchlen": 0,
    "forkpoint": "0000000000001378e25f8f992865d95aadd1b6a3cfe93889f8bee416c5c248f4",
    "status": "active"
  }

In this case block 377337 is valid but 377338 is not.

$ scc-cli invalidateblock 00000000000004d68329929c610d918efe5725d42f8c4859d624924d2c500f6f

2022-05-03T12:47:05Z InvalidChainFound: invalid block=00000000000004d68329929c610d918efe5725d42f8c4859d624924d2c500f6f  height=377338  log2_work=71.00730459  date=2022-04-29T16:12:35Z

 {
    "height": 377340,
    "hash": "00000000000022d2b554dd6e394e1cce6cecf2f7918f3deb2875a23f8352c0c9",
    "difficulty": 242509.9156163001,
    "chainwork": "000000000000000000000000000000000000000000000080a65c1b4fb5cdfae4",
    "branchlen": 3,
    "forkpoint": "0000000000001378e25f8f992865d95aadd1b6a3cfe93889f8bee416c5c248f4",
    "status": "conflicting"
  },
  {
    "height": 377338,
    "hash": "00000000000004d68329929c610d918efe5725d42f8c4859d624924d2c500f6f",
    "difficulty": 267678.7096827597,
    "chainwork": "000000000000000000000000000000000000000000000080a6548436f42afee7",
    "branchlen": 1,
    "forkpoint": "0000000000001378e25f8f992865d95aadd1b6a3cfe93889f8bee416c5c248f4",
    "status": "invalid"
  },
  {
    "height": 377337,
    "hash": "0000000000001378e25f8f992865d95aadd1b6a3cfe93889f8bee416c5c248f4",
    "difficulty": 280082.1197386601,
    "chainwork": "000000000000000000000000000000000000000000000080a6506e9428da6ea4",
    "branchlen": 0,
    "forkpoint": "0000000000001378e25f8f992865d95aadd1b6a3cfe93889f8bee416c5c248f4",
    "status": "active"
  }

Invalidating further blocks has the same effect: none.

$ sccd --version
SCC Core Daemon version v3.2.0.0-0d10c8e58

$ uname -a
Linux vps-5d014c2f 5.13.0-40-generic #45~20.04.1-Ubuntu SMP Mon Apr 4 09:38:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I tried searching for a solution but got nowhere. The only glimmer of hope came from: https://docs.bitcoincashnode.org/doc/json-rpc/invalidateblock/ https://medium.com/crownplatform/youre-forked-76f639298e84

but now I'm here.

Support suggest that a solution is being worked on. It would be great if this could be tracked.

Workaround: re-sync from scratch / from the bootstrap.

[Liquid369]

I have been working on some things related to this after the upcoming update we are planning. Could you give me some more information, maybe a debug.log. I have encountered this a few times myself and maybe we can address it in a subsequent update. The debug log would help me to see if it was a similar issue I ran into or not and we can work on it!

[KenSharp]

It's been a while since this has happened, quite possibly because I'm having to restart regularly due to the memory ballooning. It usually occurs after running for a long period of time, which I currently cannot do. Next time I get the chance I will keep the log.

[Liquid369]

We have released v3.3.0 with some memory reductions and some features related to MN Quorums processing/syncing.

If you could please test and let us know if this persists now that would be great!