Parent hash mismatch - Githubissues

Is there an existing issue?

[X] I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

[X] This is not a support question.

Description of bug

Here is Phala Network team, we got several reports that nextBlock.parentHash != block.hash the broken block is random, a user reports that he remove node db and resync and found multiple blocks have this issue.

Steps to reproduce

Here is a Khala parachain node wss://node-bugged.phala.network/ws

https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fnode-bugged.phala.network%2Fws#/explorer/query/58602 Block #58602 hash0xaf1d3ae0793192e971b591482e6722f449de7cb03626cfeaa03d1bf2e7cd9ff3 parent hash 0x505e9cc0c49bc1aeca4c3e82a661ec492dc68eb2bddddc0a9d5b289bc74e44a0

https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fnode-bugged.phala.network%2Fws#/explorer/query/58601 Block #58601 hash 0x0ce1506266a7e15262896f035b85e58cec028143d48762c820a49948eb391df8

You can see Block #58602 parent hash != Block #58601 hash

Or, try code below at https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fnode-bugged.phala.network%2Fws#/js

const hash58601 = util.u8aToHex(await api.rpc.chain.getBlockHash(58601));
console.log("#58601 hash: " + hash58601);

const hash58602 = util.u8aToHex(await api.rpc.chain.getBlockHash(58602));
console.log("#58602 hash: " + hash58602);

const header58602 = await api.rpc.chain.getHeader(hash58602);
const parentHash58602 = header58602.parentHash
console.log("#58602 parent hash: " + parentHash58602);

Output is

#58601 hash: 0x0ce1506266a7e15262896f035b85e58cec028143d48762c820a49948eb391df8
#58602 hash: 0xaf1d3ae0793192e971b591482e6722f449de7cb03626cfeaa03d1bf2e7cd9ff3
#58602 parent hash: 0x505e9cc0c49bc1aeca4c3e82a661ec492dc68eb2bddddc0a9d5b289bc74e44a0

As you can see 0x0ce1506266a7e15262896f035b85e58cec028143d48762c820a49948eb391df8 != 0x505e9cc0c49bc1aeca4c3e82a661ec492dc68eb2bddddc0a9d5b289bc74e44a0

https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fnode-bugged.phala.network%2Fws#/explorer/query/0x505e9cc0c49bc1aeca4c3e82a661ec492dc68eb2bddddc0a9d5b289bc74e44a0 shows as #58601 but different with https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fnode-bugged.phala.network%2Fws#/explorer/query/58601

So did this happen only to one user or is this reproducible? If yes, how? What database are you using?

So did this happen only to one user or is this reproducible? If yes, how? What database are you using?

We hard-coded to use RocksDB

I've got serveral reports, but I only got a sample (the DB too large, it took a week the user upload to the server), It occured total randomly, the user said he tried remove db and resync again, then got 2 broken blocks.

This sample I only found #58602 isn't right

We found this issue because our offchain service requires scan blocks one by one to build state, this issue made it stuck on the broken block

The user said: "i have to check my nodes, but it looks other blocks, i have several nodes all diff broken blocks"

So, you could not reproduce this by resyncing?

So, you could not reproduce this by resyncing?

I haven't tried resyncing, I'll try

I'm thinking this shouldn't happen, does Substrate has guard to validate blocks continuity?

I have tried 2x resyncing and now running my 3rd attempt. Previous two attempts both yielded a diff broken blocks. In the past i noticed some block crashes which caused many nodes to be stuck, requiring a restart of the node to continue syncing. Sadly i dont think we ever logged those blocks, but i wonder if these might be related.

On existing nodes i found broken blocks, but no overlapping blocks so far, e.g.:

58439 191883 783140

The bigger problem here is that when you use the node and ask for a block (e.g. 58439) it would still serve it up, but the receiver would reject it because the hash is bad. The sender in this case has no idea (while he should) that its sending bad data. There is also no way to repair the single block as far as i know, and with the issue of doing a resync is bad too ... i wonder how many nodes have a faulty DB, therewith a bad/inconsistent chain. Most users would not know this because farms have a PRB cache and workers a state file, so unless they sync from 0 they would not know ... and the receiver would only log error 500's so its even hard to understand the DB error that underlies this issue.

I have tried 2x resyncing and now running my 3rd attempt. Previous two attempts both yielded a diff broken blocks. In the past i noticed some block crashes which caused many nodes to be stuck, requiring a restart of the node to continue syncing. Sadly i dont think we ever logged those blocks, but i wonder if these might be related.

On existing nodes i found broken blocks, but no overlapping blocks so far, e.g.:

58439 191883 783140

The bigger problem here is that when you use the node and ask for a block (e.g. 58439) it would still serve it up, but the receiver would reject it because the hash is bad. The sender in this case has no idea (while he should) that its sending bad data. There is also no way to repair the single block as far as i know, and with the issue of doing a resync is bad too ... i wonder how many nodes have a faulty DB, therewith a bad/inconsistent chain. Most users would not know this because farms have a PRB cache and workers a state file, so unless they sync from 0 they would not know ... and the receiver would only log error 500's so its even hard to understand the DB error that underlies this issue.

PRB and rest of other Phala components are rely on the node data, and you can see your node data isn't correct (the reason is unknown)

The node you linked above has two blocks at this broken height. I think the problem is just that there is a broken mapping between block number and hash. I will need to dig a little bit in the code to find out how this can happen.

The node you linked above has two blocks at this broken height. I think the problem is just that there is a broken mapping between block number and hash. I will need to dig a little bit in the code to find out how this can happen.

Do you think it would be worth to implement some kind of db integrity-check command which traverses through the DB and checks this?
It could serve as a platform for more custom checks as well.

Do you think it would be worth to implement some kind of db integrity-check command which traverses through the DB and checks this?

No. I mean there seems to be some bug. This should not happen.

Yes but how do you test that the DB is consistent after your fix?
Anyway, maybe it a dumb idea.

There is a tool that dos this DB check, this is how we ended up finding the reason for the weird behaviour. Except this DB check logs we only got weird error 500's everywhere which over time got larger and larger but partially stuck software that uses the node. (would be offtopic to explain in detail).

node src/console.js debug fsck 1 25000000 --early-stop --progress --substrate-ws-endpoint ws://127.0.0.1:9944 node src/console.js debug fsck 1 25000000 --early-stop --progress --substrate-ws-endpoint ws://127.0.0.1:9945

That is how we found it. Granted i would prefer an option at the node's (API?) to trigger it locally AND have to option to repair it, cause its only 1 single block ... the blocks before that broken block and after are just fine.

Yes but how do you test that the DB is consistent after your fix?

I'm not entirely against some tool, I'm just not convinced that it needs to be part of the node. If there is stuff that can also live outside of the node in its own, self contained binary, why not put it there?

The node you linked above has two blocks at this broken height. I think the problem is just that there is a broken mapping between block number and hash. I will need to dig a little bit in the code to find out how this can happen.

Do you think it would be worth to implement some kind of db integrity-check command which traverses through the DB and checks this? It could serve as a platform for more custom checks as well.

Yeah I was considering that, but for lacking time and if I make it into Substrate we should checking more fields which I don't have enough knowledge yet. So we made an external tool to checking blocks continuity first https://github.com/Phala-Network/phala-blockchain/blob/master/scripts/js/src/console.js#L633-L673

@arkpar do you have ideas on how the storage could end up like this?

Both headers for 58601 seem to be valid, so it looks like there was a reorg, but the block number to hash mapping was not updated correctly in the DB. I'd add some logs here: https://github.com/paritytech/substrate/blob/master/client/db/src/lib.rs#L1219 and also enable sync logs to try and catch this.

Both headers for 58601 seem to be valid, so it looks like there was a reorg, but the block number to hash mapping was not updated correctly in the DB. I'd add some logs here: https://github.com/paritytech/substrate/blob/master/client/db/src/lib.rs#L1219 and also enable sync logs to try and catch this.

If its just a hash mapping in the DB that needs to be 'refreshed' would it be possible to add a method onto the node CLI or API to issue a 'refresh' of that specific block and/or range? This would solve a lot of nodes back to full integrity without the issue of resyncing weeks. That said, my 2 clean sync attempts failed (also random corrupt blocks) which i cannot clarify, its random and so far i don't see any signs of crashes or other significant errors. A 3rd clean sync is now running hoping for the best.

And to make sure: Its not hardware, its enterprise hardware with ECC+REG memory and a raid controller for storage with BBU, in a 24x 1TB SSD in raid10 and dual PSU. But as said before i also have multiple nodes on different servers (farms) having this same issue

That said, my 2 clean sync attempts failed

Did every time the same block was wrong? Or every time a different one?

That said, my 2 clean sync attempts failed

Did every time the same block was wrong? Or every time a different one?

so far all different blocks, no duplicates anywhere :/

Both headers for 58601 seem to be valid, so it looks like there was a reorg, but the block number to hash mapping was not updated correctly in the DB.

I'd add some logs here: https://github.com/paritytech/substrate/blob/master/client/db/src/lib.rs#L1219 and also enable sync logs to try and catch this.

@PHA-SYSOPS did the resync recently, but 58601 is a very ancient block, does new node sync forked blocks also? Or some other nodes propaganda forked blocks to the node?

58601 is a very ancient block

Yeah that is also something that confused me. One possible explanation could be that we synced this block from some other broken node as we sync by block numbers. However, it is still weird that this node now has the same problem, aka has this fork. This makes it even more weird. I looked over the code now multiple times and I did not yet have found any combination that could lead to fucking up the database.

58601 is a very ancient block

Yeah that is also something that confused me. One possible explanation could be that we synced this block from some other broken node as we sync by block numbers. However, it is still weird that this node now has the same problem, aka has this fork. This makes it even more weird. I looked over the code now multiple times and I did not yet have found any combination that could lead to fucking up the database.

if you need, I can give you access of the bugged node, most of our users are use the same CLI args to start their nodes

And PHA-SYSOPS is at Nederlands, so the internet access shouldn't be a problem

Hi,

Although it is not fully related, i noticed something important during the syncing monitoring:

It looks the best block is defined at startup of node (e.g. 1000)
it would get the blocks (shows as best block) up to 1000. It DOES increase target, but not used
once best block 1000 is reach, it will start to finalize, at that point it STOPS getting new blocks (best block is not increasing, target does)
once finalize is reached, it returns to step 1
so both processes never run at the same time, delaying sync significantly
if you RESTART The node, it will restart at step 1.
because the start block seems to be hardcoded at processing time (1000) and the long syncing time (about 50 days) by the time it reaches 1000, it might be 2000. So it will have to redo all these steps in such a loop likely multiple times, adding more delays overall.

Me and @jasl noticed this after we see the node is slowing down and only finalize is moving, but very slowly. At that point we restarted the node and noticed a difference in the logs block movement:

2022-08-27 09:04:16 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215762 (42 peers), best: #2184128 (0x8077…4049), finalized #614258 (0xe456…5bc9), ⬇ 9.0kiB/s ⬆ 0.4kiB/s 2022-08-27 09:04:21 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215762 (42 peers), best: #2184128 (0x8077…4049), finalized #614259 (0xe04a…1e8c), ⬇ 39.3kiB/s ⬆ 1.5kiB/s 2022-08-27 09:04:26 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215762 (42 peers), best: #2184128 (0x8077…4049), finalized #614259 (0xe04a…1e8c), ⬇ 2.5kiB/s ⬆ 0.5kiB/s 2022-08-27 09:04:31 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215762 (42 peers), best: #2184128 (0x8077…4049), finalized #614259 (0xe04a…1e8c), ⬇ 9.0kiB/s ⬆ 0.3kiB/s 2022-08-27 09:04:36 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215762 (42 peers), best: #2184128 (0x8077…4049), finalized #614259 (0xe04a…1e8c), ⬇ 7.0kiB/s ⬆ 0.4kiB/s 2022-08-27 09:04:41 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215763 (41 peers), best: #2184128 (0x8077…4049), finalized #614259 (0xe04a…1e8c), ⬇ 13.3kiB/s ⬆ 0.3kiB/s 2022-08-27 09:04:46 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215763 (41 peers), best: #2184128 (0x8077…4049), finalized #614259 (0xe04a…1e8c), ⬇ 20.5kiB/s ⬆ 0.7kiB/s 2022-08-27 09:04:51 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215764 (42 peers), best: #2184128 (0x8077…4049), finalized #614260 (0xd8da…5652), ⬇ 34.1kiB/s ⬆ 0.7kiB/s 2022-08-27 09:04:56 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215764 (42 peers), best: #2184128 (0x8077…4049), finalized #614260 (0xd8da…5652), ⬇ 28.5kiB/s ⬆ 0.4kiB/s 2022-08-27 09:05:01 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215764 (42 peers), best: #2184128 (0x8077…4049), finalized #614260 (0xd8da…5652), ⬇ 5.0kiB/s ⬆ 0.7kiB/s 2022-08-27 09:05:06 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215765 (42 peers), best: #2184128 (0x8077…4049), finalized #614260 (0xd8da…5652), ⬇ 13.5kiB/s ⬆ 0.7kiB/s 2022-08-27 09:05:11 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215765 (42 peers), best: #2184128 (0x8077…4049), finalized #614260 (0xd8da…5652), ⬇ 2.3kiB/s ⬆ 1.3kiB/s 2022-08-27 09:05:16 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215766 (42 peers), best: #2184128 (0x8077…4049), finalized #614261 (0xdbd7…08a0), ⬇ 19.8kiB/s ⬆ 0.4kiB/s 2022-08-27 09:05:21 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215766 (42 peers), best: #2184128 (0x8077…4049), finalized #614261 (0xdbd7…08a0), ⬇ 0.4kiB/s ⬆ 0.3kiB/s 2022-08-27 09:05:26 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215767 (42 peers), best: #2184128 (0x8077…4049), finalized #614261 (0xdbd7…08a0), ⬇ 25.6kiB/s ⬆ 0.6kiB/s 2022-08-27 09:05:31 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215767 (42 peers), best: #2184128 (0x8077…4049), finalized #614261 (0xdbd7…08a0), ⬇ 48.2kiB/s ⬆ 0.7kiB/s 2022-08-27 09:05:36 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215767 (42 peers), best: #2184128 (0x8077…4049), finalized #614261 (0xdbd7…08a0), ⬇ 6.5kiB/s ⬆ 0.3kiB/s 2022-08-27 09:05:41 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215767 (41 peers), best: #2184128 (0x8077…4049), finalized #614262 (0xfb6a…609b), ⬇ 53.8kiB/s ⬆ 1.2kiB/s 2022-08-27 09:05:46 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215768 (41 peers), best: #2184128 (0x8077…4049), finalized #614262 (0xfb6a…609b), ⬇ 32.6kiB/s ⬆ 0.7kiB/s 2022-08-27 09:05:51 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215768 (42 peers), best: #2184128 (0x8077…4049), finalized #614262 (0xfb6a…609b), ⬇ 43.0kiB/s ⬆ 0.9kiB/s 2022-08-27 09:05:56 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215768 (42 peers), best: #2184128 (0x8077…4049), finalized #614262 (0xfb6a…609b), ⬇ 13.2kiB/s ⬆ 0.5kiB/s 2022-08-27 09:06:01 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215769 (42 peers), best: #2184128 (0x8077…4049), finalized #614262 (0xfb6a…609b), ⬇ 30.7kiB/s ⬆ 1.1kiB/s 2022-08-27 09:06:06 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215769 (42 peers), best: #2184128 (0x8077…4049), finalized #614263 (0x621a…29a3), ⬇ 3.0kiB/s ⬆ 1.0kiB/s 2022-08-27 09:06:11 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215769 (42 peers), best: #2184128 (0x8077…4049), finalized #614263 (0x621a…29a3), ⬇ 10.2kiB/s ⬆ 0.3kiB/s 2022-08-27 09:06:16 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215769 (42 peers), best: #2184128 (0x8077…4049), finalized #614263 (0x621a…29a3), ⬇ 7.2kiB/s ⬆ 0.4kiB/s 2022-08-27 09:06:21 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215769 (42 peers), best: #2184128 (0x8077…4049), finalized #614263 (0x621a…29a3), ⬇ 24.5kiB/s ⬆ 0.7kiB/s 2022-08-27 09:06:26 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215770 (42 peers), best: #2184128 (0x8077…4049), finalized #614263 (0x621a…29a3), ⬇ 29.2kiB/s ⬆ 0.3kiB/s 2022-08-27 09:06:31 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215770 (42 peers), best: #2184128 (0x8077…4049), finalized #614264 (0xddf2…c718), ⬇ 16.5kiB/s ⬆ 25.6kiB/s 2022-08-27 09:06:36 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215771 (42 peers), best: #2184128 (0x8077…4049), finalized #614264 (0xddf2…c718), ⬇ 13.4kiB/s ⬆ 0.4kiB/s 2022-08-27 09:06:41 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215771 (42 peers), best: #2184128 (0x8077…4049), finalized #614264 (0xddf2…c718), ⬇ 6.6kiB/s ⬆ 0.4kiB/s 2022-08-27 09:06:46 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215772 (42 peers), best: #2184128 (0x8077…4049), finalized #614264 (0xddf2…c718), ⬇ 22.7kiB/s ⬆ 0.4kiB/s 2022-08-27 09:06:51 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215772 (41 peers), best: #2184128 (0x8077…4049), finalized #614264 (0xddf2…c718), ⬇ 2.0kiB/s ⬆ 0.5kiB/s 2022-08-27 09:06:56 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215772 (41 peers), best: #2184128 (0x8077…4049), finalized #614265 (0xaf26…a1b8), ⬇ 19.1kiB/s ⬆ 12.9kiB/s 2022-08-27 09:07:01 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215772 (41 peers), best: #2184128 (0x8077…4049), finalized #614265 (0xaf26…a1b8), ⬇ 17.2kiB/s ⬆ 0.6kiB/s 2022-08-27 09:07:06 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215773 (41 peers), best: #2184128 (0x8077…4049), finalized #614265 (0xaf26…a1b8), ⬇ 10.7kiB/s ⬆ 0.5kiB/s 2022-08-27 09:07:11 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215773 (42 peers), best: #2184128 (0x8077…4049), finalized #614265 (0xaf26…a1b8), ⬇ 21.5kiB/s ⬆ 0.6kiB/s 2022-08-27 09:07:16 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215773 (42 peers), best: #2184128 (0x8077…4049), finalized #614265 (0xaf26…a1b8), ⬇ 24.6kiB/s ⬆ 1.1kiB/s 2022-08-27 09:07:21 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215773 (42 peers), best: #2184128 (0x8077…4049), finalized #614341 (0x2f31…160d), ⬇ 33.0kiB/s ⬆ 0.8kiB/s 2022-08-27 09:07:26 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215773 (42 peers), best: #2184128 (0x8077…4049), finalized #614341 (0x2f31…160d), ⬇ 19.9kiB/s ⬆ 0.5kiB/s 2022-08-27 09:07:31 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215773 (42 peers), best: #2184128 (0x8077…4049), finalized #614341 (0x2f31…160d), ⬇ 11.7kiB/s ⬆ 0.4kiB/s 2022-08-27 09:07:36 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215774 (42 peers), best: #2184128 (0x8077…4049), finalized #614341 (0x2f31…160d), ⬇ 44.1kiB/s ⬆ 0.7kiB/s 2022-08-27 09:07:41 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215774 (42 peers), best: #2184128 (0x8077…4049), finalized #614341 (0x2f31…160d), ⬇ 7.6kiB/s ⬆ 0.7kiB/s 2022-08-27 09:07:46 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215775 (42 peers), best: #2184128 (0x8077…4049), finalized #614342 (0xe274…fc86), ⬇ 13.7kiB/s ⬆ 0.3kiB/s 2022-08-27 09:07:51 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215775 (42 peers), best: #2184128 (0x8077…4049), finalized #614342 (0xe274…fc86), ⬇ 1.8kiB/s ⬆ 0.4kiB/s 2022-08-27 09:07:56 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215775 (42 peers), best: #2184128 (0x8077…4049), finalized #614342 (0xe274…fc86), ⬇ 10.3kiB/s ⬆ 0.4kiB/s 2022-08-27 09:08:01 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215776 (41 peers), best: #2184128 (0x8077…4049), finalized #614342 (0xe274…fc86), ⬇ 7.1kiB/s ⬆ 0.3kiB/s 2022-08-27 09:08:06 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215776 (40 peers), best: #2184128 (0x8077…4049), finalized #614342 (0xe274…fc86), ⬇ 18.9kiB/s ⬆ 0.3kiB/s 2022-08-27 09:08:11 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215776 (40 peers), best: #2184128 (0x8077…4049), finalized #614343 (0x4a3c…8495), ⬇ 27.7kiB/s ⬆ 0.7kiB/s 2022-08-27 09:08:16 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215776 (41 peers), best: #2184128 (0x8077…4049), finalized #614343 (0x4a3c…8495), ⬇ 23.4kiB/s ⬆ 0.6kiB/s 2022-08-27 09:08:21 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215777 (41 peers), best: #2184128 (0x8077…4049), finalized #614343 (0x4a3c…8495), ⬇ 30.0kiB/s ⬆ 0.3kiB/s 2022-08-27 09:08:26 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215777 (42 peers), best: #2184128 (0x8077…4049), finalized #614343 (0x4a3c…8495), ⬇ 14.1kiB/s ⬆ 0.9kiB/s 2022-08-27 09:08:31 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215778 (42 peers), best: #2184128 (0x8077…4049), finalized #614343 (0x4a3c…8495), ⬇ 21.5kiB/s ⬆ 0.6kiB/s 2022-08-27 09:08:36 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215778 (42 peers), best: #2184128 (0x8077…4049), finalized #614344 (0x5a73…3c3e), ⬇ 37.6kiB/s ⬆ 0.5kiB/s 2022-08-27 09:08:41 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215779 (42 peers), best: #2184128 (0x8077…4049), finalized #614344 (0x5a73…3c3e), ⬇ 36.4kiB/s ⬆ 0.4kiB/s 2022-08-27 09:08:46 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215779 (42 peers), best: #2184128 (0x8077…4049), finalized #614344 (0x5a73…3c3e), ⬇ 13.0kiB/s ⬆ 0.5kiB/s 2022-08-27 09:08:51 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215779 (42 peers), best: #2184128 (0x8077…4049), finalized #614344 (0x5a73…3c3e), ⬇ 6.8kiB/s ⬆ 0.5kiB/s 2022-08-27 09:08:56 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215779 (42 peers), best: #2184128 (0x8077…4049), finalized #614344 (0x5a73…3c3e), ⬇ 13.8kiB/s ⬆ 0.3kiB/s 2022-08-27 09:09:01 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215779 (42 peers), best: #2184128 (0x8077…4049), finalized #614345 (0x55ef…8a3b), ⬇ 7.2kiB/s ⬆ 0.4kiB/s 2022-08-27 09:09:06 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215780 (42 peers), best: #2184128 (0x8077…4049), finalized #614345 (0x55ef…8a3b), ⬇ 23.0kiB/s ⬆ 0.2kiB/s 2022-08-27 09:09:11 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215780 (42 peers), best: #2184128 (0x8077…4049), finalized #614345 (0x55ef…8a3b), ⬇ 20.2kiB/s ⬆ 0.6kiB/s

As you can see, the target has moved to 2215780, but no attempts are made to increase the best block. We do see the finalizing moving (but its so slow, it will take at least 50 days to complete )

Now after the restart we see:

2022-08-27 09:23:09 [Parachain] ⚙️ Syncing 23.8 bps, target=#2215832 (32 peers), best: #2204745 (0x54dc…ff8c), finalized #614345 (0x55ef…8a3b), ⬇ 389.6kiB/s ⬆ 15.5kiB/s 2022-08-27 09:23:14 [Parachain] ⚙️ Syncing 38.8 bps, target=#2215832 (34 peers), best: #2204939 (0x5259…140e), finalized #614345 (0x55ef…8a3b), ⬇ 579.2kiB/s ⬆ 26.4kiB/s 2022-08-27 09:23:19 [Parachain] ⚙️ Syncing 43.0 bps, target=#2215832 (36 peers), best: #2205154 (0x268b…7413), finalized #614345 (0x55ef…8a3b), ⬇ 666.7kiB/s ⬆ 14.5kiB/s 2022-08-27 09:23:24 [Parachain] ⚙️ Syncing 40.0 bps, target=#2215832 (37 peers), best: #2205354 (0x8f9b…9232), finalized #614345 (0x55ef…8a3b), ⬇ 656.6kiB/s ⬆ 174.1kiB/s 2022-08-27 09:23:29 [Parachain] ⚙️ Syncing 44.2 bps, target=#2215832 (38 peers), best: #2205575 (0x15db…859e), finalized #614345 (0x55ef…8a3b), ⬇ 920.5kiB/s ⬆ 106.3kiB/s 2022-08-27 09:23:34 [Parachain] ⚙️ Syncing 45.4 bps, target=#2215832 (37 peers), best: #2205802 (0x41eb…fc24), finalized #614345 (0x55ef…8a3b), ⬇ 877.8kiB/s ⬆ 718.1kiB/s 2022-08-27 09:23:39 [Parachain] ⚙️ Syncing 44.0 bps, target=#2215833 (40 peers), best: #2206022 (0xb518…cbf6), finalized #614345 (0x55ef…8a3b), ⬇ 618.7kiB/s ⬆ 556.7kiB/s 2022-08-27 09:23:44 [Parachain] ⚙️ Syncing 41.0 bps, target=#2215833 (36 peers), best: #2206227 (0xc9e9…9938), finalized #614345 (0x55ef…8a3b), ⬇ 708.8kiB/s ⬆ 18.6kiB/s 2022-08-27 09:23:49 [Parachain] ⚙️ Syncing 38.0 bps, target=#2215835 (30 peers), best: #2206417 (0x94cf…990b), finalized #614345 (0x55ef…8a3b), ⬇ 447.1kiB/s ⬆ 2.9kiB/s 2022-08-27 09:23:54 [Parachain] ⚙️ Syncing 45.6 bps, target=#2215835 (28 peers), best: #2206645 (0x1865…cb26), finalized #614345 (0x55ef…8a3b), ⬇ 685.5kiB/s ⬆ 214.6kiB/s 2022-08-27 09:23:59 [Parachain] ⚙️ Syncing 32.2 bps, target=#2215836 (28 peers), best: #2206806 (0x4da2…81d9), finalized #614345 (0x55ef…8a3b), ⬇ 1019.9kiB/s ⬆ 2.2kiB/s 2022-08-27 09:24:04 [Parachain] ⚙️ Syncing 49.4 bps, target=#2215836 (29 peers), best: #2207053 (0xddb7…711e), finalized #614345 (0x55ef…8a3b), ⬇ 764.6kiB/s ⬆ 26.3kiB/s 2022-08-27 09:24:09 [Parachain] ⚙️ Syncing 41.0 bps, target=#2215836 (27 peers), best: #2207258 (0xcc30…1831), finalized #614345 (0x55ef…8a3b), ⬇ 674.0kiB/s ⬆ 13.0kiB/s 2022-08-27 09:24:14 [Parachain] ⚙️ Syncing 30.6 bps, target=#2215836 (26 peers), best: #2207411 (0x18c2…6629), finalized #614345 (0x55ef…8a3b), ⬇ 661.9kiB/s ⬆ 13.2kiB/s 2022-08-27 09:24:19 [Parachain] ⚙️ Syncing 37.2 bps, target=#2215836 (27 peers), best: #2207597 (0xf05f…1476), finalized #614345 (0x55ef…8a3b), ⬇ 447.8kiB/s ⬆ 14.6kiB/s 2022-08-27 09:24:24 [Parachain] ⚙️ Syncing 36.0 bps, target=#2215836 (26 peers), best: #2207777 (0x48d9…6cf2), finalized #614345 (0x55ef…8a3b), ⬇ 663.5kiB/s ⬆ 13.1kiB/s 2022-08-27 09:24:29 [Parachain] ⚙️ Syncing 42.4 bps, target=#2215837 (26 peers), best: #2207989 (0xf1c6…499a), finalized #614345 (0x55ef…8a3b), ⬇ 705.0kiB/s ⬆ 12.2kiB/s 2022-08-27 09:24:34 [Parachain] ⚙️ Syncing 33.3 bps, target=#2215837 (25 peers), best: #2208156 (0xeae4…9cc2), finalized #614345 (0x55ef…8a3b), ⬇ 896.0kiB/s ⬆ 1.8kiB/s 2022-08-27 09:24:39 [Parachain] ⚙️ Syncing 35.2 bps, target=#2215837 (26 peers), best: #2208332 (0xb3a9…5aec), finalized #614345 (0x55ef…8a3b), ⬇ 476.2kiB/s ⬆ 27.8kiB/s 2022-08-27 09:24:44 [Parachain] ⚙️ Syncing 37.0 bps, target=#2215838 (26 peers), best: #2208517 (0x5890…0848), finalized #614345 (0x55ef…8a3b), ⬇ 712.6kiB/s ⬆ 15.0kiB/s 2022-08-27 09:24:49 [Parachain] ⚙️ Syncing 36.0 bps, target=#2215838 (28 peers), best: #2208697 (0xb6d7…af94), finalized #614345 (0x55ef…8a3b), ⬇ 188.7kiB/s ⬆ 14.6kiB/s 2022-08-27 09:24:54 [Parachain] ⚙️ Syncing 36.1 bps, target=#2215838 (27 peers), best: #2208878 (0x1d97…1a3c), finalized #614345 (0x55ef…8a3b), ⬇ 1.8kiB/s ⬆ 2.0kiB/s 2022-08-27 09:24:59 [Parachain] ⚙️ Syncing 37.4 bps, target=#2215838 (29 peers), best: #2209065 (0x3805…dba4), finalized #614345 (0x55ef…8a3b), ⬇ 1.3MiB/s ⬆ 14.9kiB/s 2022-08-27 09:25:04 [Parachain] ⚙️ Syncing 38.8 bps, target=#2215838 (29 peers), best: #2209259 (0xe427…7fea), finalized #614345 (0x55ef…8a3b), ⬇ 1.1MiB/s ⬆ 23.5kiB/s 2022-08-27 09:25:09 [Parachain] ⚙️ Syncing 33.0 bps, target=#2215838 (29 peers), best: #2209424 (0x5118…a996), finalized #614345 (0x55ef…8a3b), ⬇ 684.2kiB/s ⬆ 3.2kiB/s 2022-08-27 09:25:14 [Parachain] ⚙️ Syncing 33.4 bps, target=#2215838 (29 peers), best: #2209591 (0x9626…64da), finalized #614345 (0x55ef…8a3b), ⬇ 224.5kiB/s ⬆ 1.1kiB/s 2022-08-27 09:25:19 [Parachain] ⚙️ Syncing 34.2 bps, target=#2215839 (29 peers), best: #2209762 (0xc8b0…017a), finalized #614345 (0x55ef…8a3b), ⬇ 899.4kiB/s ⬆ 27.4kiB/s 2022-08-27 09:25:24 [Parachain] ⚙️ Syncing 30.4 bps, target=#2215840 (31 peers), best: #2209914 (0x1b0e…892b), finalized #614345 (0x55ef…8a3b), ⬇ 232.6kiB/s ⬆ 26.1kiB/s 2022-08-27 09:25:29 [Parachain] ⚙️ Syncing 31.4 bps, target=#2215842 (30 peers), best: #2210071 (0x9613…8a50), finalized #614345 (0x55ef…8a3b), ⬇ 676.7kiB/s ⬆ 15.6kiB/s 2022-08-27 09:25:34 [Parachain] ⚙️ Syncing 36.5 bps, target=#2215842 (30 peers), best: #2210254 (0x6823…d7aa), finalized #614345 (0x55ef…8a3b), ⬇ 846.8kiB/s ⬆ 14.1kiB/s 2022-08-27 09:25:39 [Parachain] ⚙️ Syncing 37.1 bps, target=#2215842 (34 peers), best: #2210440 (0xd7b8…b977), finalized #614345 (0x55ef…8a3b), ⬇ 507.8kiB/s ⬆ 11.6kiB/s 2022-08-27 09:25:44 [Parachain] ⚙️ Syncing 40.9 bps, target=#2215842 (33 peers), best: #2210645 (0x8b0f…1960), finalized #614345 (0x55ef…8a3b), ⬇ 831.3kiB/s ⬆ 2.6kiB/s 2022-08-27 09:25:49 [Parachain] ⚙️ Syncing 39.0 bps, target=#2215843 (33 peers), best: #2210840 (0x02e4…3761), finalized #614345 (0x55ef…8a3b), ⬇ 522.8kiB/s ⬆ 1.8kiB/s 2022-08-27 09:25:54 [Parachain] ⚙️ Syncing 42.8 bps, target=#2215843 (34 peers), best: #2211054 (0xe773…4ecc), finalized #614345 (0x55ef…8a3b), ⬇ 899.3kiB/s ⬆ 2.4kiB/s 2022-08-27 09:25:59 [Parachain] ⚙️ Syncing 33.6 bps, target=#2215844 (30 peers), best: #2211222 (0x5bac…e95d), finalized #614345 (0x55ef…8a3b), ⬇ 457.7kiB/s ⬆ 1.7kiB/s 2022-08-27 09:26:04 [Parachain] ⚙️ Syncing 35.2 bps, target=#2215844 (33 peers), best: #2211398 (0x1466…62f3), finalized #614345 (0x55ef…8a3b), ⬇ 596.9kiB/s ⬆ 1.2kiB/s 2022-08-27 09:26:09 [Parachain] ⚙️ Syncing 39.6 bps, target=#2215844 (31 peers), best: #2211596 (0x61ce…6402), finalized #614345 (0x55ef…8a3b), ⬇ 807.0kiB/s ⬆ 1.1kiB/s 2022-08-27 09:26:14 [Parachain] ⚙️ Syncing 37.2 bps, target=#2215844 (31 peers), best: #2211782 (0x574d…dddf), finalized #614345 (0x55ef…8a3b), ⬇ 649.2kiB/s ⬆ 1.4kiB/s 2022-08-27 09:26:19 [Parachain] ⚙️ Syncing 41.2 bps, target=#2215845 (31 peers), best: #2211988 (0xe59a…8cf8), finalized #614345 (0x55ef…8a3b), ⬇ 685.8kiB/s ⬆ 1.8kiB/s 2022-08-27 09:26:24 [Parachain] ⚙️ Syncing 39.8 bps, target=#2215845 (31 peers), best: #2212187 (0x9cb8…9358), finalized #614345 (0x55ef…8a3b), ⬇ 678.5kiB/s ⬆ 1.3kiB/s 2022-08-27 09:26:29 [Parachain] ⚙️ Syncing 36.8 bps, target=#2215845 (33 peers), best: #2212371 (0xc32c…8f3c), finalized #614345 (0x55ef…8a3b), ⬇ 905.1kiB/s ⬆ 2.9kiB/s 2022-08-27 09:26:34 [Parachain] ⚙️ Syncing 35.2 bps, target=#2215845 (33 peers), best: #2212547 (0x2a9b…47d8), finalized #614345 (0x55ef…8a3b), ⬇ 457.2kiB/s ⬆ 26.3kiB/s 2022-08-27 09:26:39 [Parachain] ⚙️ Syncing 45.0 bps, target=#2215846 (35 peers), best: #2212772 (0x062d…ff91), finalized #614345 (0x55ef…8a3b), ⬇ 866.0kiB/s ⬆ 17.7kiB/s 2022-08-27 09:26:44 [Parachain] ⚙️ Syncing 38.0 bps, target=#2215847 (32 peers), best: #2212962 (0x41d0…de82), finalized #614345 (0x55ef…8a3b), ⬇ 513.7kiB/s ⬆ 26.2kiB/s 2022-08-27 09:26:49 [Parachain] ⚙️ Syncing 43.4 bps, target=#2215847 (30 peers), best: #2213179 (0xd7cc…9932), finalized #614345 (0x55ef…8a3b), ⬇ 902.1kiB/s ⬆ 39.2kiB/s 2022-08-27 09:26:54 [Parachain] ⚙️ Syncing 38.2 bps, target=#2215848 (31 peers), best: #2213370 (0x0561…e74b), finalized #614345 (0x55ef…8a3b), ⬇ 256.5kiB/s ⬆ 16.1kiB/s 2022-08-27 09:26:59 [Parachain] ⚙️ Syncing 42.6 bps, target=#2215848 (33 peers), best: #2213583 (0xafbe…51bc), finalized #614345 (0x55ef…8a3b), ⬇ 521.2kiB/s ⬆ 13.8kiB/s 2022-08-27 09:27:04 [Parachain] ⚙️ Syncing 38.2 bps, target=#2215848 (36 peers), best: #2213774 (0xae1e…5d48), finalized #614345 (0x55ef…8a3b), ⬇ 1017.3kiB/s ⬆ 11.3kiB/s 2022-08-27 09:27:09 [Parachain] ⚙️ Syncing 22.0 bps, target=#2215848 (36 peers), best: #2213884 (0xa867…33b4), finalized #614345 (0x55ef…8a3b), ⬇ 341.1kiB/s ⬆ 1.1kiB/s 2022-08-27 09:27:14 [Parachain] ⚙️ Syncing 34.4 bps, target=#2215849 (36 peers), best: #2214056 (0xa95e…c23a), finalized #614345 (0x55ef…8a3b), ⬇ 66.2kiB/s ⬆ 1.0kiB/s 2022-08-27 09:27:19 [Parachain] ⚙️ Syncing 39.6 bps, target=#2215849 (32 peers), best: #2214254 (0x279f…4b7e), finalized #614345 (0x55ef…8a3b), ⬇ 8.4kiB/s ⬆ 1.1kiB/s 2022-08-27 09:27:24 [Parachain] ⚙️ Syncing 34.1 bps, target=#2215849 (33 peers), best: #2214425 (0xc0a0…cb53), finalized #614345 (0x55ef…8a3b), ⬇ 10.7kiB/s ⬆ 0.9kiB/s 2022-08-27 09:27:29 [Parachain] ⚙️ Syncing 43.4 bps, target=#2215851 (26 peers), best: #2214642 (0x35ca…cb63), finalized #614345 (0x55ef…8a3b), ⬇ 2.7kiB/s ⬆ 1.9kiB/s 2022-08-27 09:27:34 [Parachain] ⚙️ Syncing 35.2 bps, target=#2215852 (24 peers), best: #2214818 (0x2748…480d), finalized #614345 (0x55ef…8a3b), ⬇ 8.1kiB/s ⬆ 0.8kiB/s 2022-08-27 09:27:39 [Parachain] ⚙️ Syncing 40.0 bps, target=#2215852 (27 peers), best: #2215018 (0x5f00…e69f), finalized #614345 (0x55ef…8a3b), ⬇ 5.0kiB/s ⬆ 1.1kiB/s 2022-08-27 09:27:44 [Parachain] ⚙️ Syncing 36.8 bps, target=#2215852 (27 peers), best: #2215202 (0x91af…f923), finalized #614345 (0x55ef…8a3b), ⬇ 8.1kiB/s ⬆ 2.0kiB/s 2022-08-27 09:27:49 [Parachain] ⚙️ Syncing 40.7 bps, target=#2215852 (27 peers), best: #2215406 (0xa3c0…8d4c), finalized #614345 (0x55ef…8a3b), ⬇ 9.7kiB/s ⬆ 1.4kiB/s 2022-08-27 09:27:54 [Parachain] ⚙️ Syncing 27.0 bps, target=#2215852 (27 peers), best: #2215541 (0x7d0d…f634), finalized #614345 (0x55ef…8a3b), ⬇ 5.3kiB/s ⬆ 2.6kiB/s 2022-08-27 09:27:59 [Parachain] ⚙️ Syncing 34.7 bps, target=#2215852 (28 peers), best: #2215715 (0x9532…8511), finalized #614345 (0x55ef…8a3b), ⬇ 8.0kiB/s ⬆ 3.4kiB/s

Now notice that the best block is moving, but finalized is NOT. As the system has plenty of resources it could go a lot faster, it feels like an logic issue. After some time, the best hold reaches the target height:

2022-08-27 09:29:49 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215862 (26 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 50.3kiB/s ⬆ 4.2kiB/s 2022-08-27 09:29:54 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215863 (26 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 7.5kiB/s ⬆ 1.0kiB/s 2022-08-27 09:29:59 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215863 (28 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 22.4kiB/s ⬆ 5.6kiB/s 2022-08-27 09:30:04 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215863 (28 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 9.8kiB/s ⬆ 2.3kiB/s 2022-08-27 09:30:09 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215864 (34 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 12.9kiB/s ⬆ 3.2kiB/s 2022-08-27 09:30:14 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215864 (35 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 2.8kiB/s ⬆ 1.0kiB/s 2022-08-27 09:30:19 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215864 (39 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 7.3kiB/s ⬆ 1.0kiB/s 2022-08-27 09:30:24 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215864 (40 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 4.3kiB/s ⬆ 1.0kiB/s 2022-08-27 09:30:29 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215864 (41 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 8.2kiB/s ⬆ 1.0kiB/s 2022-08-27 09:30:34 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215865 (42 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 4.3kiB/s ⬆ 1.2kiB/s 2022-08-27 09:30:39 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215865 (38 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 8.0kiB/s ⬆ 1.6kiB/s 2022-08-27 09:30:44 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215866 (38 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 5.8kiB/s ⬆ 1.0kiB/s 2022-08-27 09:30:49 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215866 (28 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 12.2kiB/s ⬆ 1.7kiB/s 2022-08-27 09:30:54 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215866 (27 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 1.0kiB/s ⬆ 0.9kiB/s 2022-08-27 09:30:59 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215867 (23 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 2.9kiB/s ⬆ 1.0kiB/s 2022-08-27 09:31:04 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215867 (26 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 5.9kiB/s ⬆ 1.1kiB/s 2022-08-27 09:31:09 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215867 (27 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 2.4kiB/s ⬆ 0.6kiB/s 2022-08-27 09:31:14 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215867 (27 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 4.1kiB/s ⬆ 0.6kiB/s

As you can see, syncing has stopped and you clearly can see the target still moving ... but the best block seems to be 'defined' as 2215856, which was more or less the block during startup of the node. Currently finalizing is not moving, but it looks like the system is doing some database work (not logged, but stracing shows heavy I/O on reading (97%) and some writes (3%). As this is the point of writing this reply, i cannot show the finalized moving after restart, but i have checked logs from other nodes in our farms which show similar behavior.

after the long I/O wait it just jumped:

2022-08-27 09:46:44 [Parachain] ⚙️ Syncing 0.0 bps, target=#2215925 (38 peers), best: #2215856 (0x5b5b…d52c), finalized #614345 (0x55ef…8a3b), ⬇ 3.3kiB/s ⬆ 62.6kiB/s 2022-08-27 09:46:49 [Parachain] ⚙️ Syncing 1.8 bps, target=#2215926 (17 peers), best: #2215866 (0xa47a…b43e), finalized #2215856 (0x5b5b…d52c), ⬇ 253.1kiB/s ⬆ 107.7kiB/s

where it was still 45 days to go, the restart seems to have boosted it to near instant completion, but then just before completion i get:

2022-08-27 09:46:47 [Parachain] ✨ Imported #2215857 (0x81b6…c218) 2022-08-27 09:46:47 [Parachain] ✨ Imported #2215858 (0xe0cc…9878) 2022-08-27 09:46:48 [Relaychain] ✨ Imported #14188023 (0x7adf…09fb) 2022-08-27 09:46:49 [Parachain] Block import error: Potential long-range attack: block not in finalized chain. 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xd13c2542e1012106babc75b692d5940c8585b4fe57fab87ed14ceb5c94100572: consensus error: Import failed: Potential long-range attack: block not in finalized chain. 2022-08-27 09:46:49 [Parachain] ✨ Imported #2215859 (0xac7e…a8f3) 2022-08-27 09:46:49 [Parachain] ✨ Imported #2215859 (0x7bcb…1010) 2022-08-27 09:46:49 [Parachain] ✨ Imported #2215860 (0xaf39…7bf7) 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x775ee9f1594e8bfdf9ba5523e8896dafa3443ab3e570919142898dea5453f601: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x77ca9c1541fe645b87c465e6449a3940e2cf3399d93d31b83dc8951c6af28f15: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xaa420935508f51ab270f847c5ac0acf71924c313cb6a006a1712f9640f35d554: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xda75b81a008e40ff01e80647bd6a57728da3693a6d998104048c8accce6ae3e5: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x0c37701dc9a3ce1ea409b970f57d04571a6107f5d5ba1a6ad6f5abcd93c28b3c: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xed1a3eda5b49a63107d5b4775a5258d677c91bd12f9980a037adb86fa5c245f1: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xb159486b14ce5bc07648860ebe022f7c5d76074111fc7d717b61c2e1ccd2bb4d: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x958c7ef75584aa7a1e5ab250500b086a2c548116bf30b6a3fd63aa73a2bf47df: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x9207b6a506074bc6c5ee1650e81cf8d42326aaef64c294a7e6d752bb5ea035c8: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x94feb4a0084d48626c180f01685aedc6f289995d51f22e87b7c30279d5cd579e: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xc0cb1bd2b09d51663d8a47a61cac448517975472a914eac5988bb573db4e841e: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xbf3e8940394099e46f3c858c8d7906cb7d603510785901a29c5a0fc3881d11e9: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xde60f12d2ea18dbaa74348957a130476904a15f401a439f53b806fd602a4fe47: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xc466c7c4e211140e24489792b061b6e2d0fd69b700b6406481571a72610ca7aa: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xe9392d08888359c78de75c4661121016bd25a71358d8640449026ec074804191: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xfd6026eb6d91e0c217ea7caf8d4799dd7dce222e49f170d4324c873887893d1d: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xd07ef5200eb207846be2c3c9a5e4f1e18378010181cb0ad6a720a03728b12d42: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x24c57c9bbda9884e20cec77a7339bcd878ebada00bffaf765fd78c5320844dd2: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x2c5ba7f7654f634f05885038bd9725d13c900141e6436eb81c1c623605a23d12: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xd3465915da82686cc1534058f4f23e6377214fada460fc554351ae6e16a0b6cc: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x0cc822983487e057b164a08293a7add6670d35426c9694a4f3a0082f6eb963cf: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x6bbd2cccdf52d273a2369bf8829b27f0f34e52b6891a9567a9dbf8ffcdb1e347: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xfd9a20c91996bb1e0d541e57034cd6759a56e41dc8ea30bdfe8e3cd44bf8a2a3: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x0244a766e686a4836d4c3d4aeb18c984db08a2052bfd55e778834a110367041a: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x0db3b0221cb0a3cffce9fad0e277bc8f2fe2e7a2247c6d9d60d1b8a04b35bcfb: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xf0866d29703e1ce4cb78854f2e5f44a1465ff376f162c85c73260626cf89a702: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x0dbd0dbbf829d54cc3366ec922446b945a4374c756dc76c9e8749044937defdb: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x00252e005e8733e64db22b7e724c712c06cdc7481bf727f3141ebd0b1b1a1847: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xa5d9981aa9c4830de96d9d36a28044973075b477e860455d28201bf25d036f9e: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x6f0e1dd61d60429f4070ee4facc82f6519e30166b0657a236870b21ef5195379: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x6623709f65a18ddd9b9386629d7d116f7a6a4e26e3e832b3bbb4c899d10b70be: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x5fc75540f00e8a94cf408d701c280faea6e4d65310971fc19d0c8f71bbf90d22: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xa131e9fbad037a1dda985a2594c11f4c5b754e91a0aa860e1a29eb1c8412fa99: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xdb9187db49e62eaf7e2717ebd9075741127d6b5705d11008735712ef04a2e159: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x6897495ca333c9c693332840ba613b036cb1ea17452c06940d7bc4b5ea86809c: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x13e2e345d9fc2c17fa186121a7e6624523670fe6a566459414f63cd03351a535: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xd6ca69e894ca3a4daa0eb7de6a51b967cb715c39520a367bb83ade0f2beb8aaf: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xe19c9da967036342c934ae2ecef174d0f6f7e0f5e8ea3cbe361c6bd112496033: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x90d7782cb13b9bbe6d4e5d75f2e805c47f87e9f0360eb475fda4d7c6f95faaa3: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x435bb9be0867c941457c9ff37428ba124a6238dc92c9de25f1529419a8bf59d9: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x2a7812adf59beb45f2953b100962c899aef5c9b20fbdc4c66339cf8ea65ae8f9: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xc2b2244e185b906ae01328672ec8391a4e172a94fc9891b4992d7a92f5224ad1: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x6f1f6e20b4786496e9afbc34885ac02dc01cc2ce418817f4915c54b5d41a28cd: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xb372fd2574b1aa9b205206d052da66fa43877f9fe30241e84378a60b280f58e5: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x192a280f510d837f169983563bd1851be708eaf1fbb53cb7276842112062de55: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xcc3e89604077c58c98f04691810297b11eaca82984fc09cef369e9945db91255: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xb6a84a20d6e27eea23de3e9eeea9aa733138b0d9e99f64f428040ec5416cf51a: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x9288def41abba8032371202ea8539c1c433a5dabd27863872ba569b0a5e57fec: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xbbd54ed84e825b6d71809b3233f4cdeeaf6df923c6dc1c556af068b9973a8920: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x9bc181ab3786b122a442b4727310944588084be05b15632224e8f0576fa13db6: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xc42741ae7898f7e12a146c1a5c2c74ef4b99345d130aac043232edf4d9fa695a: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x110eaf4c0f9f762c0f23d1061cd063d452998c0a1cea45b8f05fee5e9d2164a0: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xb787cb7dc3a491006a152d7310d25bb12044a1288fe7e4f51b26e2485651e637: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x5fe0deb88b149d30aa710db8243c14c873393a386e5a44ba6213030a438e4347: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x22dab55772e205ff419817f87a8c940f13b2abc0559e0bfd02019f8bb278c172: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xc65cbcf96bb8de1ddacc5d87055d7eab885452e0d789986c0aa7b62c1756b551: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0xdf7f6b068ee0adf5812d969a18ab0efaa09530c9a835624e1c5f50a91324c6e1: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x3671a1d4e21ab84e51f29b01bae0bae06c7dd719a6b2cf0147488c594fdf8ea5: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x4408fb52446cc27ab1507d7762a3e9c35068f939537a000b7c5bfc008f4124f4: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x818921dac794f38780a89d93ed1de9e4915cccfc3ddcf50c0e1037370690945b: block has an unknown parent 2022-08-27 09:46:49 [Parachain] 💔 Error importing block 0x91d35fff1544787e48d83fcb82340629e361d3a462c276e34bd81278935db16c: block has an unknown parent 2022-08-27 09:46:49 [Parachain] ✨ Imported #2215861 (0x1dfa…0246) 2022-08-27 09:46:49 [Parachain] ✨ Imported #2215862 (0x9e5b…372b) 2022-08-27 09:46:49 [Parachain] ✨ Imported #2215863 (0xc8f3…5ed3) 2022-08-27 09:46:49 [Parachain] ✨ Imported #2215864 (0x1a1b…46ca) 2022-08-27 09:46:49 [Parachain] ✨ Imported #2215864 (0xeb67…906b) 2022-08-27 09:46:49 [Parachain] ✨ Imported #2215866 (0xa47a…b43e) 2022-08-27 09:46:49 [Parachain] ⚙️ Syncing 1.8 bps, target=#2215926 (17 peers), best: #2215866 (0xa47a…b43e), finalized #2215856 (0x5b5b…d52c), ⬇ 253.1kiB/s ⬆ 107.7kiB/s 2022-08-27 09:46:50 [Relaychain] 💤 Idle (28 peers), best: #14188023 (0x7adf…09fb), finalized #14188018 (0x5373…3f85), ⬇ 87.4kiB/s ⬆ 141.1kiB/s 2022-08-27 09:46:54 [Relaychain] ✨ Imported #14188024 (0x85d6…0374) 2022-08-27 09:46:54 [Parachain] 💤 Idle (26 peers), best: #2215927 (0x24c3…72bb), finalized #2215856 (0x5b5b…d52c), ⬇ 43.2kiB/s ⬆ 109.5kiB/s 2022-08-27 09:46:55 [Relaychain] 💤 Idle (28 peers), best: #14188024 (0x85d6…0374), finalized #14188020 (0xc620…0d0b), ⬇ 137.7kiB/s ⬆ 321.3kiB/s 2022-08-27 09:46:56 [Parachain] ✨ Imported #2215928 (0xe5dd…e3c9) 2022-08-27 09:46:59 [Parachain] 💤 Idle (24 peers), best: #2215927 (0x24c3…72bb), finalized #2215856 (0x5b5b…d52c), ⬇ 31.0kiB/s ⬆ 56.5kiB/s 2022-08-27 09:47:00 [Relaychain] 💤 Idle (28 peers), best: #14188024 (0x85d6…0374), finalized #14188021 (0x6760…344d), ⬇ 346.5kiB/s ⬆ 280.5kiB/s

i am now running FSCK script, but this kind of weird behavior might cause database problems. In addition i noticed that during the syncing (loop as described above) the FSCK script does not start, as if the WSS/API is stuck too. I have logged the exact moment of restarting, so if that block (range) is broken we might have some clues here. I will follow up with FSCK results when i have them.

to be clear, the parent errors ONLY shown as above, no other errors before that. AFter that there is 1 error that i did not past here:

2022-08-27 09:59:10 [Relaychain] 💔 Error importing block 0x0f4ff91aa4d9db290016b7562f76753cd4786c0ca92815a4aaa3763b54e982fc: consensus error: Import failed: Parent block of 0x0f4f…82fc has no associated weight

This concludes all logs lines with 'error' or 'Error'

Update: the 3rd node sync was successful, both DB checks are completed with no errors. The only change is i removed the --state-cache-size 0 argument. I will retry another system with this argument and see what happens.

paritytech / substrate

Parent hash mismatch #12086

Is there an existing issue?

Experiencing problems? Have you tried our Stack Exchange first?

Description of bug

Steps to reproduce