The block has been deleted and terminated, but windowspost always displays an error

lbj2004032 commented 2 years ago

Checklist

[X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
[X] This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
[X] This is not a new feature request. If it is, please file a feature request instead.
[X] This is not an enhancement request. If it is, please file a improvement suggestion instead.
[X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
[X] I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
[X] I did not make any code changes to lotus.

Lotus component

[ ] lotus daemon - chain sync
[ ] lotus miner - mining and block production
[ ] lotus miner/worker - sealing
[ ] lotus miner - proving(WindowPoSt)
[ ] lotus miner/market - storage deal
[ ] lotus miner/market - retrieval deal
[ ] lotus miner/market - data transfer
[ ] lotus client
[ ] lotus JSON-RPC API
[ ] lotus message management (mpool)
[ ] Other

Lotus Version

lotus-miner version 1.15.3+mainnet+git.2f6a38302.dirty

Describe the Bug

The block has been deleted and terminated, but windowspost always displays an error Because I operated snap up before and saved data with data cap, two block errors occurred. Then I deleted the block, but the block status displayed that it was deleted. However, windowspost always had an error, and the other block was in terminatewait status the issue reference, https://github.com/filecoin-project/lotus/issues/8805

Logging Information

lotus@miner-172:~$ lotus-miner sectors status  0
SectorID:       0
Status:         Removed
CIDcommD:       baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
CIDcommR:       bagboea4b5abcamr7vlaomd5edxfi2b4jdb3ehcxt7jud2jmkl5u3rqkxmzwv7ctm
Ticket:         460dc1d168af80a992f61e1461b21541f0f1afbb6e02c027cef9336c6d864992
TicketH:        1504349
Seed:           dded69baec6e7aea129874acb3949fc2abbef1c1ee693e7f501cec01c440e5a9
SeedH:          1506267
Precommit:      bafy2bzacedpxgtkdbzdux7u73qvcws4rjw7uzozawzpg5d3xokpge456cyyra
Commit:         bafy2bzacebwefx5gff6pc4vqw645ecfkoprqgkjyo4xgmdudaaee67mwsaotc
Deals:          [6661791]
Retries:        0

2321.   2022-06-04 11:10:44 +0800 CST:  [event;sealing.SectorFinalizeFailed]    {"User":{}}
        finalize sector: storage call error 0: %!w(allocate local sector for fetching: couldn't find a suitable path for a sector [Hostname: miner-172.70.1.12])
2322.   2022-06-04 11:11:44 +0800 CST:  [event;sealing.SectorRemove]    {"User":{}}
2323.   2022-06-04 11:11:44 +0800 CST:  [event;sealing.SectorRetryFinalize]     {"User":{}}
2324.   2022-06-04 11:11:44 +0800 CST:  [event;sealing.SectorRetryFinalize]     {"User":{}}
2325.   2022-06-04 11:11:44 +0800 CST:  [event;sealing.SectorRemoved]   {"User":{}}
2326.   2022-06-04 11:11:44 +0800 CST:  [event;sealing.SectorRemoved]   {"User":{}}
2327.   2022-06-04 11:11:44 +0800 CST:  [event;sealing.SectorRemoved]   {"User":{}}
2328.   2022-06-04 11:11:44 +0800 CST:  [error;<nil>]   state machine error: %!s(<nil>)
2329.   2022-06-04 11:11:44 +0800 CST:  [event;sealing.SectorRemoved]   {"User":{}}
2330.   2022-06-04 11:11:44 +0800 CST:  [error;<nil>]   state machine error: %!s(<nil>)
2331.   2022-06-04 11:13:14 +0800 CST:  [event;sealing.SectorTerminate] {"User":{}}
2332.   2022-06-04 11:13:14 +0800 CST:  [error;<nil>]   state machine error: %!s(<nil>)
2333.   2022-06-05 11:12:54 +0800 CST:  [event;sealing.SectorRemove]    {"User":{}}
2334.   2022-06-05 11:12:54 +0800 CST:  [error;<nil>]   state machine error: %!s(<nil>)
2335.   2022-06-05 11:38:38 +0800 CST:  [event;sealing.SectorTerminate] {"User":{}}
2336.   2022-06-05 11:38:38 +0800 CST:  [error;<nil>]   state machine error: %!s(<nil>)
2337.   2022-06-06 13:31:33 +0800 CST:  [event;sealing.SectorTerminate] {"User":{}}
2338.   2022-06-06 13:31:33 +0800 CST:  [error;<nil>]   state machine error: %!s(<nil>)
2339.   2022-06-06 13:38:23 +0800 CST:  [error;<nil>]   state machine error: %!s(<nil>)
2340.   2022-06-13 10:18:52 +0800 CST:  [error;<nil>]   state machine error: %!s(<nil>)
2341.   2022-06-13 10:24:04 +0800 CST:  [event;sealing.SectorRemove]    {"User":{}}
2342.   2022-06-13 10:24:04 +0800 CST:  [error;<nil>]   state machine error: %!s(<nil>)
2343.   2022-06-13 10:24:32 +0800 CST:  [event;sealing.SectorTerminate] {"User":{}}
2344.   2022-06-13 10:24:32 +0800 CST:  [error;<nil>]   state machine error: %!s(<nil>)

Repo Steps

Run '...'
Do '...'
See error '...' ...

Reiers commented 2 years ago

Hi @lbj2004032

Thanks for the report. Looks like you are running Lotus with custom code or adjustment .dirty Please upgrade to stock lotus, make clean and make all again !

When you say block do you mean sector, ?

If so and the sectors are stuck in faulty state and like you stated deleted, use lotus-shed to remove them.

cd lotus/
make lotus-shed
./lotus-shed sectors terminate --really-do-it 0

Let me know how it goes, closing since its related to #8805

lbj2004032 commented 2 years ago

./lotus-shed sectors terminate --really-do-it 0 it works

filecoin-project / lotus