search

filecoin-project / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
2.83k stars 1.25k forks source link

Running out of disk space causes strange sync errors #2760

Closed whyrusleeping closed 2 years ago

whyrusleeping commented 4 years ago

My disk is at 100% and badger is predictably falling to pieces:

sync status:
worker 0:
        Base:   [bafy2bzacebpvpjs5723roonfaalrsm7v47ucypvz2oo63kg4ooz2qbz7a4ybi]
        Target: [bafy2bzacedmhxite7td7n6efndytvhayiprbbbefmridymydn3qmibeokztl4] (1667)
        Height diff:    47
        Stage: error
        Height: 1667
        Elapsed: 2.00291657s
        Error: chain contained block marked previously as bad ([bafy2bzacedmhxite7td7n6efndytvhayiprbbbefmridymydn3qmibeokztl4], bafy
2bzacec5bdhmedv3tvcvxrpqjw3yhrifr7jd35qdc44p7dsrnf655zylcy) (reason: linked to bafy2bzacedkzgsz7myj5yicnk3h2nx2anbzjfovxlebbdua5or3kg
nnlnpszi caused by: [bafy2bzacecmdnzxysnfhp2onoktpgy253kcspokmzbme4eudljzgras464h3o] 1 error occurred:
        * block had invalid messages:
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock.func3
        /home/why/code/lotus/chain/sync.go:695
  - failed to load base state tree:
    github.com/filecoin-project/lotus/chain.(*Syncer).checkBlockMessages
        /home/why/code/lotus/chain/sync.go:970
  - failed to load hamt node:
    github.com/filecoin-project/specs-actors/actors/util/adt.AsMap
        /home/why/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.8.6/actors/util/adt/map.go:41
  - Invalid read: Len: 1558 read at:[9942:9952]
    github.com/dgraph-io/badger/v2.(*valueLog).Read
        /home/why/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200716180832-3ab515320794/value.go:1570
    github.com/dgraph-io/badger/v2.(*Item).yieldItemValue
        /home/why/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200716180832-3ab515320794/iterator.go:170
    github.com/dgraph-io/badger/v2.(*Item).ValueCopy
        /home/why/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200716180832-3ab515320794/iterator.go:127
    github.com/ipfs/go-ds-badger2.(*txn).get
        /home/why/go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:512
    github.com/ipfs/go-ds-badger2.(*Datastore).Get
        /home/why/go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:302
    github.com/ipfs/go-ds-measure.(*measure).Get
        /home/why/go/pkg/mod/github.com/ipfs/go-ds-measure@v0.1.0/measure.go:198
    github.com/ipfs/go-datastore/keytransform.(*Datastore).Get
        /home/why/go/pkg/mod/github.com/ipfs/go-datastore@v0.4.4/keytransform/keytransform.go:47
    github.com/ipfs/go-ipfs-blockstore.(*blockstore).Get
        /home/why/go/pkg/mod/github.com/ipfs/go-ipfs-blockstore@v1.0.0/blockstore.go:122
    github.com/ipfs/go-ipfs-blockstore.(*idstore).Get
        /home/why/go/pkg/mod/github.com/ipfs/go-ipfs-blockstore@v1.0.0/idstore.go:57
    github.com/ipfs/go-ipfs-blockstore.(*arccache).Get
        /home/why/go/pkg/mod/github.com/ipfs/go-ipfs-blockstore@v1.0.0/arc_cache.go:118
    github.com/ipfs/go-ipfs-blockstore.(*bloomcache).Get
        /home/why/go/pkg/mod/github.com/ipfs/go-ipfs-blockstore@v1.0.0/bloom_cache.go:159
    github.com/ipfs/go-ipfs-blockstore.(*idstore).Get
        /home/why/go/pkg/mod/github.com/ipfs/go-ipfs-blockstore@v1.0.0/idstore.go:57
    github.com/ipfs/go-ipld-cbor.(*BasicIpldStore).Get
        /home/why/go/pkg/mod/github.com/ipfs/go-ipld-cbor@v0.0.5-0.20200428170625-a0bd04d3cbdf/store.go:44
    github.com/ipfs/go-hamt-ipld.LoadNode
        /home/why/go/pkg/mod/github.com/ipfs/go-hamt-ipld@v0.1.1/hamt.go:166
    github.com/filecoin-project/specs-actors/actors/util/adt.AsMap
        /home/why/go/pkg/mod/github.com/filecoin-project/specs-actors@v0.8.6/actors/util/adt/map.go:39
    github.com/filecoin-project/lotus/chain/state.LoadStateTree
        /home/why/code/lotus/chain/state/statetree.go:128
    github.com/filecoin-project/lotus/chain.(*Syncer).checkBlockMessages
        /home/why/code/lotus/chain/sync.go:968
    github.com/filecoin-project/lotus/chain.(*Syncer).ValidateBlock.func3
        /home/why/code/lotus/chain/sync.go:694
    github.com/Gurpartap/async.Err.func1
        /home/why/go/pkg/mod/github.com/!gurpartap/async@v0.0.0-20180927173644-4f7f499dd9ee/error.go:29
    runtime.goexit
        /usr/local/lib/go/src/runtime/asm_amd64.s:1373
RobQuistNL commented 4 years ago

Related: https://github.com/filecoin-project/lotus/issues/1958 :)

weaming commented 3 years ago

Same situation :(

Reiers commented 2 years ago

This is solved and outdated. Closing ticket.

Still experiencing sync problems? Please see: https://lotus.filecoin.io/docs/set-up/chain-management/ or ask for help in Discussions or in #fil-lotus-help.