search

openethereum / parity-ethereum

The fast, light, and robust client for Ethereum-like networks.
Other
6.81k stars 1.68k forks source link

Unable to do 'db reset' #11118

Closed ghost closed 4 years ago

ghost commented 4 years ago

My Parity Ropsten node has been running for days. It is fully synchronised and I do not prune. Now I'd like to update from 2.5.8 to 2.5.9 but it does not sync anymore:

2019-10-02 15:12:49 UTC Syncing #6504673 0xbcb9…8e95     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #6504673    4/25 peers   3 MiB chain 8 KiB db 0 bytes queue 4 KiB sync  RPC:  0 conn,    2 req/s,   25 µs
2019-10-02 15:12:52 UTC Stage 5 block verification failed for #6504674 (0x2573…bba3)
Error: Error(Block(InvalidStateRoot(Mismatch { expected: 0x4c09fcd74e1251bcf814e2f5ab88df8a65447a69092097deb6f27724ada3d0c8, found: 0x31cf4de42da0c47f592d756d266bb9e8879a7dfdf8744df3aa5d84b9c3555b9d })), State { next_error: None, backtra
ce: InternalBacktrace { backtrace: None } })
2019-10-02 15:12:52 UTC
Bad block detected: Error(Block(InvalidStateRoot(Mismatch { expected: 0x4c09fcd74e1251bcf814e2f5ab88df8a65447a69092097deb6f27724ada3d0c8, found: 0x31cf4de42da0c47f592d756d266bb9e8879a7dfdf8744df3aa5d84b9c3555b9d })), State { next_error:
None, backtrace: InternalBacktrace { backtrace: None } })

My understanding is that I need to 'rewind' to block 6485846 because of the Istanbul fork. I tried to do parity db reset --chain ropsten 2 (or any other number). I always get this message:

====================

stack backtrace:
   0:     0x555e84a5570d - <no info>
   1:     0x555e84a527be - <no info>
   2:     0x555e84a5257e - <no info>
   3:     0x555e84a97df4 - <no info>
   4:     0x555e84a97b6e - <no info>
   5:     0x555e84ab4645 - <no info>
   6:     0x555e84aba58b - <no info>
   7:     0x555e84ac07f5 - <no info>
   8:     0x555e84516a09 - <no info>
   9:     0x555e83bcbebe - <no info>
  10:     0x555e84de3e9d - <no info>
  11:     0x555e84bcb7df - <no info>
  12:     0x555e84bc8340 - <no info>
  13:     0x555e84cc28f2 - <no info>
  14:     0x555e84bc6ecd - <no info>
  15:     0x7fc9cf38709a - <no info>
  16:     0x555e839fee78 - <no info>
  17:                0x0 - <no info>

Thread 'main' panicked at 'best_block_hash was fetched from db; block_header_data should exist in db; qed', src/libcore/option.rs:1036

This is a bug. Please report it at:

    https://github.com/paritytech/parity-ethereum/issues/new

Aborted

How could I fix this?

jam10o-new commented 4 years ago

(cc @seunlanlege?)

seunlanlege commented 4 years ago

Looks like you have the latest block hash but not the block header. I'm not sure how that's possible but it might have something to do with this.

Bad block detected: Error(Block(InvalidStateRoot(Mismatch { expected: 0x4c09fcd74e1251bcf814e2f5ab88df8a65447a69092097deb6f27724ada3d0c8, found: 0x31cf4de42da0c47f592d756d266bb9e8879a7dfdf8744df3aa5d84b9c3555b9d })), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })

I will look into why that happens, but for now I would suggest a re-sync with 2.5.9 if you can.

quietnan commented 4 years ago

Colleague of @g574 here.

The issue with resyncing our Ropsten parity node is that it's a full archiving node with traceing enabled and we also need it that way. Resyncing takes several weeks. Since we can only start testing in preparation for the mainnet launch after that sync is finished, this likely leads to us being late for production. Plus, we also consider the Istanbul activation on Ropsten as a test of the Istanbul activation on mainnet, where a resync after a failed upgrade would take several months... So just giving up on the upgrade and resyncing can unfortunately not be an option for us.

jam10o-new commented 4 years ago

You've mentioned that reset results in the same panic regardless of the value you give it - have you confirmed (just want to verify) that this is the case for values that would directly reset you to prior to the fork block? My concern is that if you were to reset to a block post-fork that your node wouldn't consider valid it may no longer be present.

ghost commented 4 years ago

@joshua-mir Yes, we've tried that. Here's what happens when I try with a high number:

parity db reset --chain ropsten 18827
2019-10-03 06:56:51 UTC Configured for Ropsten Testnet using Ethash engine
2019-10-03 06:56:51 UTC unable to get mut ref for engine for shutdown.
Attempting to reset to block with pruned state

(18827 would be the exact number we need to rewind by.)

seunlanlege commented 4 years ago

@g574 @quietnan

can you try to download this custom build of parity which would provide better logs of the problem

https://drive.google.com/file/d/1bRpwyMdNPy9oFrAr8dN0zFUlhjUqdkG5/view?usp=sharing

ghost commented 4 years ago

I have decided to resync the whole database.