Closed LordKurono closed 1 year ago
I noticed the same issue when scratching from sync. I thought it was just a POKT issue the entire time, but now that you mention it, I was also using the latest staging branch. Interesting that RC v0.9.2 is working just fine for you, definitely worthy sanity-checking before we upgrade to V0.10.0.
When syncing from scratch, I also received wrong Block.Header.AppHash.
. I also tried a snapshot around block ~60,000 and also received wrong Block.Header.AppHash.
as an error, but this one might've been caused due to some other unrelated testing.
FYI: We are not on RC 0.10.0 yet on mainnet. But i'm not sure why that would make the syncing fail. Best case - this is not an issue, worst case - it was flagged before the network was upgraded. @LordKurono thanks for the report
@PoktBlade you likely have more experience synching nodes from scratch so would appreciate your help here:
Does this happen every time or ephemerally?
Have you hit it in the past?
What do you recommend we sanity check?
Which snapshot are you using?
Would you be willing to take on this investigation?
@PoktBlade you likely have more experience synching nodes from scratch so would appreciate your help here:
- Does this happen every time or ephemerally?
- Have you hit it in the past?
- What do you recommend we sanity check?
- Which snapshot are you using?
- Would you be willing to take on this investigation?
I actually don't have too much experience syncing from scratch, however, I do know it's pretty important that syncing from a point of time still works.
When syncing from scratch, pretty much every time.
🤔
No, I haven't actually synced from scratch in a very long long time. I did hear reports this was an issue before in the past, but the details are really fuzzy.
Q1
: @msmania @okdas Have either of you seen this?
Have either of you seen this?
Try to sync from not so far snapshot (i.e. block 85xxx) and see if it crashes, or if this is specific to scratch only or really historical snapshots only.
R1
: @PoktBlade Please give it a shot and report back! You can look at the snapshots @Andrew-Pohl's team has set up in #1565
The only thing that really changed with the state machine explicitly in V0.10.0 was this #1534 (files). So we can start there while debugging the app state of the validator as well. The other thing I think we should look into is the upgraded app version if that has any significance when syncing.
Looked at the code and I don't fully understand why it would impact it given that we haven't upgraded the protocol but trust your judgment. R2
: Please report back once you've given it a shot!
link.us1.storjshare.io/raw/jwfbmq6ar3vsyzeqkconsiz24sja/pocket-public-blockchains/pocket-network-data-0026-rc-0.6.3.6.tar was the snapshot I was trying. I was trying an old snapshot for some other testing purposes unrelated to this issue.
Repeating comment above, take a look at #1565 to see if it helps.
I can help triage the issue a bit more for you and report back here. I.E - I can try to revert the above PR and see if the issue still persists. But preferably if it is due to a recent change in v0.10.0, let's get the right code owners to apply a fix once triaged.
R2
: Can you revert (i.e. hard reset) to 410e12c3b40106e6e9c98edd4f17ed1234a7d435
and see if the same issue occurs?
R3
: Can you rm 0aa12e71a33fca29c8e8ad5f60dcccae775a8045
and see if the same issue occurs?
Thanks again for your help in advance! Seeing PNF (@jacklaing) for visibility to know that community members are helping debug this.
Legend:
triaged and fixed in https://github.com/pokt-network/pocket-core/pull/1566
Resolved in @PoktBlade's fix!
Describe the bug Starting a new pocket node on version RC-0.10.0 using this command:
pocket start --seeds="7c0d7ec36db6594c1ffaa99724e1f8300bbd52d0@seed1.mainnet.pokt.network:26662,cdcf936d70726dd724e0e6a8353d8e5ba5abdd20@seed2.mainnet.pokt.network:26663,74b4322a91c4a7f3e774648d0730c1e610494691@seed3.mainnet.pokt.network:26662,b3235089ff302c9615ba661e13e601d9d6265b15@seed4.mainnet.pokt.network:26663" --mainnet
Crashes after processing the first block
Using version RC-0.9.2 yields the expected behaviors and the node syncs.
Expected behavior Node syncs from block 1 to TOP of the chain