search

consensus-shipyard / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
7 stars 3 forks source link

TestMirBasic_AllNodesMiningWithMessaging is flaky #85

Closed dnkolegov closed 10 months ago

dnkolegov commented 1 year ago

https://app.circleci.com/pipelines/github/consensus-shipyard/lotus/185/workflows/01722f08-8dff-493a-91f9-1cf78e61f1f3/jobs/1368

2023-02-24T16:41:30.577Z    INFO    mir-consensus   mir/manager.go:283  manager received error:could not process PassiveModule (app) events: app restore state error: RestoreState: validator t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i failed to wait for next block 20: validator t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i context canceled while waiting for a snapshot {"validator": "t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i"}
2023-02-24T16:41:30.577Z    INFO    mir-consensus   mir/manager.go:337  Mir manager stopping    {"validator": "t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i"}
2023-02-24T16:41:30.577Z    ERROR   mir-manager mir/logger.go:33    Interceptor: Interceptor returned with error.   {"err": "interceptor stopped at caller request", "nodeID": "t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i"}
2023-02-24T16:41:30.577Z    INFO    mir-consensus   mir/manager.go:350  Interceptor closed  {"validator": "t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i"}
2023-02-24T16:41:30.577Z    INFO    mir-consensus   mir/manager.go:354  Network transport stopped   {"validator": "t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i"}
2023-02-24T16:41:30.578Z    INFO    mir-consensus   mir/manager.go:358  Mir manager stopped {"validator": "t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i"}
2023-02-24T16:41:30.578Z    INFO    mir-consensus   runtime/panic.go:838    Mir manager serve stopped   {"validator": "t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i"}
panic: validator t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i consensus error: could not process PassiveModule (app) events: app restore state error: RestoreState: validator t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i failed to wait for next block 20: validator t1r2zq5e4gcisq6xhwtcpmkhsf7iak6i76tec4x4i context canceled while waiting for a snapshot

goroutine 689508 [running]:
github.com/filecoin-project/lotus/chain/consensus/mir.(*Manager).Serve(0xc00e712000, {0x6450898?, 0xc013a924c0?})
    /home/circleci/project/chain/consensus/mir/manager.go:285 +0x15ae
github.com/filecoin-project/lotus/chain/consensus/mir.Mine({0x6450898, 0xc013a924c0}, {{0xc008e5e660?, 0xc00541c790?}}, {0x6466ce0?, 0xc01874d380?}, {0x6480cc0?, 0xc0050a4b00?}, {0x6449300, 0xc001c09850}, ...)
    /home/circleci/project/chain/consensus/mir/mine.go:28 +0x17d
github.com/filecoin-project/lotus/itests/kit.(*Ensemble).BeginMirMiningWithDelayForFaultyNodes.func1()
    /home/circleci/project/itests/kit/ensemble.go:1162 +0x25c
golang.org/x/sync/errgroup.(*Group).Go.func1()
    /home/circleci/go/pkg/mod/golang.org/x/sync@v0.0.0-20220907140024-f12130a52804/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
    /home/circleci/go/pkg/mod/golang.org/x/sync@v0.0.0-20220907140024-f12130a52804/errgroup/errgroup.go:72 +0xa5
dnkolegov commented 1 year ago 
2023-02-24T16:48:38.847Z    INFO    mir-consensus   runtime/panic.go:838    Mir manager serve stopped   {"validator": "t1yb6gz6jgmcqna672qsswwouzajavojqyk3vk45a"}
panic: validator t1yb6gz6jgmcqna672qsswwouzajavojqyk3vk45a consensus error: could not process PassiveModule (app) events: app restore state error: RestoreState: validator t1yb6gz6jgmcqna672qsswwouzajavojqyk3vk45a failed to wait for next block 36: validator t1yb6gz6jgmcqna672qsswwouzajavojqyk3vk45a context canceled while waiting for a snapshot

goroutine 693635 [running]:
github.com/filecoin-project/lotus/chain/consensus/mir.(*Manager).Serve(0xc004157c20, {0x6450898?, 0xc0093c3000?})
    /home/circleci/project/chain