I can not win any block for more than ten days after I changed miner owner address

lbj2004032 commented 2 years ago

Checklist

[X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
[X] This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
[X] This is not a new feature request. If it is, please file a feature request instead.
[X] This is not an enhancement request. If it is, please file a improvement suggestion instead.
[X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
[X] I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
[X] I did not make any code changes to lotus.

Lotus component

[ ] lotus daemon - chain sync
[X] lotus miner - mining and block production
[ ] lotus miner/worker - sealing
[X] lotus miner - proving(WindowPoSt)
[ ] lotus miner/market - storage deal
[ ] lotus miner/market - retrieval deal
[ ] lotus miner/market - data transfer
[ ] lotus client
[ ] lotus JSON-RPC API
[ ] lotus message management (mpool)
[ ] Other

Lotus Version

lotus-miner version 1.14.1+mainnet+git.d2dca7b21.dirty

Describe the Bug

I can not win any block for more than ten days after I changed miner owner address,even when I change the newaddres back to the old address. How can I solve this problem? cmd: lotus-miner actor set-owner --really-do-it && lotus-miner actor set-owner --really-do-it

miner info Enabled subsystems (from miner API): [Mining Sealing SectorStorage Markets] Enabled subsystems (from markets API): [Mining Sealing SectorStorage Markets] Chain: [sync ok] [basefee 174.982 pFIL] Miner: **** (32 GiB sectors) Power: 985 Ti / 16.6 Ei (0.0057%) Raw: 985.5 TiB / 16.02 EiB (0.0059%) Committed: 996.4 TiB Proving: 985.5 TiB (32 GiB Faulty, 0.00%)

Logging Information

lotus-miner actor set-owner --really-do-it <new address> <old address> && lotus-miner actor set-owner --really-do-it <new address> <new address>

Repo Steps

lotus-miner actor set-owner --really-do-it && lotus-miner actor set-owner --really-do-it

Kubuxu commented 2 years ago

I can confirm the probability that no blocks were won in the 10 day period is highly unlikely. Logs surrounding possible won blocks and winning post would help with debugging.

More than likely the change of the owner address is not the cause of it as the owner address is not used in any block production steps.

lbj2004032 commented 2 years ago

22-05-06T21:44:00.796+0800 ERROR events events/events_called.go:228 chain trigger (@H 1784602, triggered @ 1784608) failed: looking for publish deal message bafy2bzacebw6x2lr7f7um7p3eb346gbtfc3p5lx6hb6paewsnde7skfa7xb5k: search msg failed: failed to load message: blockstore: block not found 2022-05-06T21:44:00.831+0800 ERROR events events/events_called.go:228 chain trigger (@H 1784602, triggered @ 1784608) failed: looking for publish deal message bafy2bzacecxx5n6jon57kotviezw5dxqj5hgsllav2kx7rkztgknsaqhbp236: search msg failed: failed to load message: blockstore: block not found 2022-05-06T21:44:00.863+0800 ERROR events events/events_called.go:228 chain trigger (@H 1784602, triggered @ 1784608) failed: looking for publish deal message bafy2bzacedgumwderbhqho7uvuihtvh7cavaf5c47fss3h5upebz6ukkyugfy: search msg failed: failed to load message: blockstore: block not found 2022-05-06T21:44:00.897+0800 ERROR events events/events_called.go:228 chain trigger (@H 1784602, triggered @ 1784608) failed: looking for publish deal message bafy2bzacebjhfsp7kr2nvkbwhddpyzjrsuanjvsqpcg2nxbxvso6zdwlc7qfk: search msg failed: failed to load message: blockstore: block not found 2022-05-06T21:44:00.932+0800 ERROR events events/events_called.go:228 chain trigger (@H 1784602, triggered @ 1784608) failed: looking for publish deal message bafy2bzacebjhfsp7kr2nvkbwhddpyzjrsuanjvsqpcg2nxbxvso6zdwlc7qfk: search msg failed: failed to load message: blockstore: block not found 2022-05-06T21:44:00.966+0800 ERROR events events/events_called.go:228 chain trigger (@H 1784602, triggered @ 1784608) failed: looking for publish deal message bafy2bzacebjhfsp7kr2nvkbwhddpyzjrsuanjvsqpcg2nxbxvso6zdwlc7qfk: search msg failed: failed to load message: blockstore: block not found 2022-05-06T21:44:01.000+0800 ERROR events events/events_called.go:228 chain trigger (@H 1784602, triggered @ 1784608) failed: looking for publish deal message bafy2bzacebdfca5cxhv4ljkq7mxduhh5ec4tamuxru4g4yrcf4uhqxbpoewm4: search msg failed: failed to load message: blockstore: block not found 2022-05-06T21:44:01.033+0800 ERROR events events/events_called.go:228 chain trigger (@H 1784602, triggered @ 1784608) failed: looking for publish deal message bafy2bzacechbtizjthpjugai4yfmtr5x6v3xxfnzfn4kdgfdih532b5ubpw46: search msg failed: failed to load message: blockstore: block not found 2022-05-06T21:44:01.252+0800 WARN advmgr sector-storage/sched_worker.go:215 failed to check worker session {"error": "RPC client error: sendRequest failed: Post \"http://172.16.0.74:5341/rpc/v0\": dial tcp 172.16.0.74:5341: connect: connection refused"} 2022-05-06T21:44:01.325+0800 WARN advmgr sector-storage/sched_worker.go:215 failed to check worker session {"error": "RPC client error: sendRequest failed: Post \"http://172.16.0.72:5343/rpc/v0\": dial tcp 172.16.0.72:5343: connect: connection refused"} 2022-05-06T21:44:02.728+0800 WARN advmgr sector-storage/sched_worker.go:215 failed to check worker session {"error": "RPC client error: sendRequest failed: Post \"http://172.16.0.73:5341/rpc/v0\": dial tcp 172.16.0.73:5341: connect: connection refused"} 2022-05-06T21:44:02.831+0800 WARN advmgr sector-storage/sched_worker.go:215 failed to check worker session {"error": "RPC client error: sendRequest failed: Post \"http://172.16.0.71:3741/rpc/v0\": dial tcp 172.16.0.71:3741: connect: connection refused"} 2022-05-06T21:44:03.076+0800 WARN advmgr sector-storage/sched_worker.go:215 failed to check worker session {"error": "RPC client error: sendRequest failed: Post \"http://172.16.0.71:5343/rpc/v0\": dial tcp 172.16.0.71:5343: connect: connection refused"} 2022-05-06T21:44:04.052+0800 WARN advmgr sector-storage/sched_worker.go:215 failed to check worker session {"error": "RPC client error: sendRequest failed: Post \"http://172.16.0.74:5343/rpc/v0\": dial tcp 172.16.0.74:5343: connect: connection refused"} 2022-05-06T21:44:04.165+0800 WARN advmgr sector-storage/sched_worker.go:215 failed to check worker session {"error": "RPC client error: sendRequest failed: Post \"http://172.16.0.74:5342/rpc/v0\": dial tcp 172.16.0.74:5342: connect: connection refused"} 2022-05-06T21:44:04.853+0800 WARN advmgr sector-storage/sched_worker.go:215 failed to check worker session {"error": "RPC client error: sendRequest failed: Post \"http://172.16.0.73:5343/rpc/v0\": dial tcp 172.16.0.73:5343: connect: connection refused"} 2022-05-06T21:44:05.203+0800 WARN advmgr sector-storage/sched_worker.go:215 failed to check worker session {"error": "RPC client error: sendRequest failed: Post \"http://172.16.0.71:5341/rpc/v0\": dial tcp 172.16.0.71:5341: connect: connection refused"} 2022-05-06T21:44:06.019+0800 WARN advmgr sector-storage/sched_worker.go:215 failed to check worker session {"error": "RPC client error: sendRequest failed: Post \"http://172.16.0.73:5342/rpc/v0\": dial tcp 172.16.0.73:5342: connect: connection refused"} 2022-05-06T21:44:06.020+0800 INFO miner miner/miner.go:479 completed mineOne {"tookMilliseconds": 5, "forRound": 1784609, "baseEpoch": 1784608, "baseDeltaSeconds": 6, "nullRounds": 0, "lateStart": false, "beaconEpoch": 1880453, "lookbackEpochs": 900, "networkPowerAtLookback": "19154089014571892736", "minerPowerAtLookback": "1083538495569920", "isEligible": true, "isWinner": false, "error": null}

lbj2004032 commented 2 years ago

mineOne {"tookMilliseconds": 5, "forRound": 1784609 ，“baseEpoch”：1784608，“baseDeltaSeconds”：6，“nullRounds”：0，“lateStart”：false，“beaconEpoch”：1880453，“lookbackEpochs”：900，“networkPowerAtLookback”：“19154089014571892736”，“minerPowerAtLookback”：“ 1083538495569920", "isEligible": true, "isWinner": false, "error": null}

lbj2004032 commented 2 years ago

events/events_called.go:228 chain trigger (@H 1784608, triggered @ 1784614) failed: looking for publish deal message bafy2bzacedgumwderbhqho7uvuihtvh7cavaf5c47fss3h5upebz6ukkyugfy: search msg failed: failed to load message: blockstore: block not found 2022-05-06T21:47:00.663+0800 ERROR events events/events_called.go:228 chain trigger (@H 1784608, triggered @ 1784614) failed: looking for publish deal message bafy2bzacebjhfsp7kr2nvkbwhddpyzjrsuanjvsqpcg2nxbxvso6zdwlc7qfk: search msg failed: failed to load message: blockstore: block not found 2022-05-06T21:47:00.684+0800 ERROR events events/events_called.go:228 chain trigger (@H 1784608, triggered @ 1784614) failed: looking for publish deal message bafy2bzacea7mj4hgqoicnizees22wbznwfuyx6dccqcnksepxfenia4pclaec: search msg failed: failed to load message: blockstore: block not found

RobQuistNL commented 2 years ago

please look for "isWinner": true and surrounding logs

rjan90 commented 2 years ago

Hey @lbj2004032!

What is the update on this? Have you won any blocks since this issue where created? As Rob pointed out above we would need logs that look for "isWinner": true and surrounding logs to look further into this.

Can you also specify if there some upgrades/downtime during the timeline you outlined?

github-actions[bot] commented 2 years ago

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 24 hours.

lbj2004032 commented 2 years ago

Hey @lbj2004032!

What is the update on this? Have you won any blocks since this issue where created? As Rob pointed out above we would need logs that look for "isWinner": true and surrounding logs to look further into this.

Can you also specify if there some upgrades/downtime during the timeline you outlined?

I won a new block the next day after I changed owner address to the old address

lbj2004032 commented 2 years ago

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 24 hours.

Our owner address is f3... but we want a f1... address, so I tried exceute < lotus-miner actor set-owner --really-do-it && lotus-miner actor set-owner --really-do-it> to change the miner owner address

rjan90 commented 2 years ago

Thanks for the updated information!

Although the probability that no blocks were won in the 10 day period is highly unlikely given your power in the network. We are certain that the lack of won blocks is not related to the change of an owner address.

I will move this issue to a discussion so we can further dig into why it happened, most notably we would need more logs for periode, and information about if there where some upgrades/downtime during the timeline you outlined?

filecoin-project / lotus