search

filecoin-project / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
2.83k stars 1.25k forks source link

0.6.x broken #3654

Closed karalabe closed 2 years ago

karalabe commented 4 years ago

After sync, it ends up with a corrupted database. Resyned multiple times, same error, different blocks.

2020-09-08T12:38:39.389+0300    ERROR   chainstore  store/store.go:694  get message get failed: bafy2bzacect7vrufx7igyynm2duwlgkioelqpxg32v74fqc5vayj2j37tymrc: blockstore: block not found
2020-09-08T12:38:32.515+0300    INFO    chain   chain/sync.go:689   block validation    {"took": 0.027462411, "height": "41282", "age": 13052.515601222}
2020-09-08T12:38:32.515+0300    ERROR   chain   chain/sync.go:1429  failed to validate tipset: validating block bafy2bzaceaqjtoiasi3gus4d77ooofwz3iorfbykb7wdu6nhsgord74veieyw:
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateTipSet.func1
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:604
  - get tipsetstate(41282, [bafy2bzacecn46qn2rsxx5frvaofn2qg4nuykxwnslzhhp6zgfl4gz5cs5x6l2]) failed:
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:773
  - error handling state forks:
    github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).ApplyBlocks
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/stmgr.go:206
  - loading state tree failed:
    github.com/filecoin-project/lotus/chain/stmgr.UpgradeFaucetBurnRecovery
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/forks.go:104
  - load state tree:
    github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).WithStateTree
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/read.go:50
  - failed to load hamt node:
    github.com/filecoin-project/specs-actors/actors/util/adt.AsMap
        /home/karalabe/work/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.7/actors/util/adt/map.go:41
  - blockstore: block not found
2020-09-08T12:38:32.515+0300    ERROR   chain   chain/sync_manager.go:389   sync error: collectChain failed:
    github.com/filecoin-project/lotus/chain.(*Syncer).Sync
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:554
  - collectChain syncMessages:
    github.com/filecoin-project/lotus/chain.(*Syncer).collectChain
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:1613
  - message processing failed:
    github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState.func1
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:1430
  - validating block bafy2bzaceaqjtoiasi3gus4d77ooofwz3iorfbykb7wdu6nhsgord74veieyw:
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateTipSet.func1
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:604
  - get tipsetstate(41282, [bafy2bzacecn46qn2rsxx5frvaofn2qg4nuykxwnslzhhp6zgfl4gz5cs5x6l2]) failed:
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:773
  - error handling state forks:
    github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).ApplyBlocks
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/stmgr.go:206
  - loading state tree failed:
    github.com/filecoin-project/lotus/chain/stmgr.UpgradeFaucetBurnRecovery
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/forks.go:104
  - load state tree:
    github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).WithStateTree
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/read.go:50
  - failed to load hamt node:
    github.com/filecoin-project/specs-actors/actors/util/adt.AsMap
        /home/karalabe/work/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.7/actors/util/adt/map.go:41
  - blockstore: block not found

Keeps printing in loop

karalabe commented 4 years ago

Same fail on 0.5.10 too

jleni commented 4 years ago

We are seeing a similar issue

karalabe commented 4 years ago

Btw, this seems to be some data race. I resynced today about 7 times now, always fails at a different block. Currently I'm playing "sync 200 blocks, copy datadir, repeat".

karalabe commented 4 years ago

No, seems to always fail on block 41285 now for me.

2020-09-08T16:36:49.340+0300    ERROR   statetree   state/statetree.go:130  loading hamt node bafy2bzacecnw4ml2ylcb7bqyjqvfeearfuzj5vdg6ayc3uzzi3sppmnsz5lli failed: failed to load hamt node: blockstore: block not found
CooperStudy commented 3 years ago

After sync, it ends up with a corrupted database. Resyned multiple times, same error, different blocks.

2020-09-08T12:38:39.389+0300  ERROR   chainstore  store/store.go:694  get message get failed: bafy2bzacect7vrufx7igyynm2duwlgkioelqpxg32v74fqc5vayj2j37tymrc: blockstore: block not found
2020-09-08T12:38:32.515+0300  INFO    chain   chain/sync.go:689   block validation    {"took": 0.027462411, "height": "41282", "age": 13052.515601222}
2020-09-08T12:38:32.515+0300  ERROR   chain   chain/sync.go:1429  failed to validate tipset: validating block bafy2bzaceaqjtoiasi3gus4d77ooofwz3iorfbykb7wdu6nhsgord74veieyw:
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateTipSet.func1
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:604
  - get tipsetstate(41282, [bafy2bzacecn46qn2rsxx5frvaofn2qg4nuykxwnslzhhp6zgfl4gz5cs5x6l2]) failed:
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:773
  - error handling state forks:
    github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).ApplyBlocks
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/stmgr.go:206
  - loading state tree failed:
    github.com/filecoin-project/lotus/chain/stmgr.UpgradeFaucetBurnRecovery
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/forks.go:104
  - load state tree:
    github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).WithStateTree
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/read.go:50
  - failed to load hamt node:
    github.com/filecoin-project/specs-actors/actors/util/adt.AsMap
        /home/karalabe/work/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.7/actors/util/adt/map.go:41
  - blockstore: block not found
2020-09-08T12:38:32.515+0300  ERROR   chain   chain/sync_manager.go:389   sync error: collectChain failed:
    github.com/filecoin-project/lotus/chain.(*Syncer).Sync
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:554
  - collectChain syncMessages:
    github.com/filecoin-project/lotus/chain.(*Syncer).collectChain
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:1613
  - message processing failed:
    github.com/filecoin-project/lotus/chain.(*Syncer).syncMessagesAndCheckState.func1
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:1430
  - validating block bafy2bzaceaqjtoiasi3gus4d77ooofwz3iorfbykb7wdu6nhsgord74veieyw:
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateTipSet.func1
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:604
  - get tipsetstate(41282, [bafy2bzacecn46qn2rsxx5frvaofn2qg4nuykxwnslzhhp6zgfl4gz5cs5x6l2]) failed:
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/sync.go:773
  - error handling state forks:
    github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).ApplyBlocks
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/stmgr.go:206
  - loading state tree failed:
    github.com/filecoin-project/lotus/chain/stmgr.UpgradeFaucetBurnRecovery
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/forks.go:104
  - load state tree:
    github.com/filecoin-project/lotus/chain/stmgr.(*StateManager).WithStateTree
        /home/karalabe/work/src/github.com/filecoin-project/lotus/chain/stmgr/read.go:50
  - failed to load hamt node:
    github.com/filecoin-project/specs-actors/actors/util/adt.AsMap
        /home/karalabe/work/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.7/actors/util/adt/map.go:41
  - blockstore: block not found

Keeps printing in loop

do you solve the problem，I meet the problems too

CooperStudy commented 3 years ago

how to solve the problem

Reiers commented 2 years ago

Hi

Thanks for the report. This is fixed in the latest release. Please upgrade Lotus.

Link to latest tags here.

Closing ticket.

If you are still experiencing problems, open a new ticket or ask for help in our Discussions forum.