lotus-miner proving compute windowpost during lotus-miner storage cleanup crashes lotus-miner

hyunmoon commented 1 year ago

Checklist

[X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
[X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
[X] I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
[ ] I did not make any code changes to lotus.

Lotus component

[ ] lotus daemon - chain sync
[ ] lotus fvm/fevm - Lotus FVM and FEVM interactions
[ ] lotus miner/worker - sealing
[X] lotus miner - proving(WindowPoSt/WinningPoSt)
[ ] lotus JSON-RPC API
[ ] lotus message management (mpool)
[ ] Other

Lotus Version

Daemon:  1.21.0-rc1+mainnet+git.3f8a41cfe+api1.5.0
Local: lotus-miner version 1.21.0-rc1+mainnet+git.3f8a41cfe

Repro Steps

Runlotus-miner storage cleanup Before cleanup is finished, run lotus-miner proving compute window-post

Describe the Bug

lotus-miner proving compute windowpost during lotus-miner storage cleanup crashes lotus-miner.

lotus-miner proving compute window-post 12
Took 2.167362687s
ERROR: RPCConnectionError

Logging Information

rjan90 commented 1 year ago

When you say crashes here, do you mean that the RPCConnection dies, or that the lotus-miner process crashes. Its unclear from the logging here.

Do youy have any logs from the time this was happening? In the meantime we will try to reproduce this error

hyunmoon commented 1 year ago

Both cleanup and proving compute ended with ERROR: RPCConnectionError and lotus-miner process crashed at the same time.

hyunmoon commented 1 year ago

I could not find anything special before the process crashed, so I didn't back up the log unfortunately.

rjan90 commented 1 year ago

Can you check if there where any reports in dmesg/systemd?

hyunmoon commented 1 year ago

Today, lotus-miner crashed again despite I downgraded everything (node, miner, worker) back to v1.20.4

No dmesg logs at the time. This time I backed up the log.

2023-03-29T14:47:02.515+0900    WARN    fsutil  fsutil/dealloc_linux.go:22  could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.516+0900    WARN    fsutil  fsutil/dealloc_linux.go:22  could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.520+0900    WARN    fsutil  fsutil/dealloc_linux.go:22  could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.521+0900    WARN    fsutil  fsutil/dealloc_linux.go:22  could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.522+0900    WARN    fsutil  fsutil/dealloc_linux.go:22  could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.524+0900    WARN    fsutil  fsutil/dealloc_linux.go:22  could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.526+0900    WARN    fsutil  fsutil/dealloc_linux.go:22  could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.528+0900    WARN    fsutil  fsutil/dealloc_linux.go:22  could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.529+0900    WARN    fsutil  fsutil/dealloc_linux.go:22  could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.529+0900    WARN    fsutil  fsutil/dealloc_linux.go:22  could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:04.010+0900    WARN    sectors pipeline/fsm.go:799 sector 325680 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325680},"ProofType":8},"aPQfsubT3uUQF3k3wZEzr8gK7r0X8hVpa4a9EREROf0=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqfd5ml5tztcfcydskrstmjhyn55bby6xb3ya27chssnadtdhfxefy"}}]]]), got ()
2023-03-29T14:47:04.012+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325680), waiting 59.987845509s before retrying
2023-03-29T14:47:07.013+0900    WARN    sectors pipeline/fsm.go:799 sector 326271 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326271},"ProofType":8},"uSVj9leeFVVrj2FwsiVPdeX43wx7Kir/inoqPKDzn+U=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqjhd4qyf3rpbnvpvlpg4iilo5dhbke23k7vw7jvxz2dzc37jld2mi"}}]]]), got ()
2023-03-29T14:47:07.015+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(326271), waiting 59.984417492s before retrying
2023-03-29T14:47:10.026+0900    INFO    miner   miner/miner.go:478  completed mineOne   {"tookMilliseconds": 22, "forRound": 2725415, "baseEpoch": 2725414, "baseDeltaSeconds": 10, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2821260, "lookbackEpochs": 900, "networkPowerAtLookback": "22416530710540058624", "minerPowerAtLookback": "19488925475143680", "isEligible": true, "isWinner": false, "error": null}
2023-03-29T14:47:13.006+0900    WARN    sectors pipeline/fsm.go:799 sector 325994 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325994},"ProofType":8},"ccJjop/rEPT9trRMBsUpaqRjSRvWNmcIC6Y2IKe5wpQ=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqglwaz47wfphh6swm53fk6w6235a66aokaudfiac3jng4aza2toea"}}]]]), got ()
2023-03-29T14:47:13.007+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325994), waiting 59.992171923s before retrying
2023-03-29T14:47:16.008+0900    WARN    sectors pipeline/fsm.go:799 sector 326190 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326190},"ProofType":8},"2Gxdy7CA3UujfxzdaATXW/bMgZ5rmcqKZ8icX27SvgE=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqnxeqn7ov4pmzp42fci4av6fqxpcm5aue2mymtwclgbv5mq2h6ini"}}]]]), got ()
2023-03-29T14:47:16.009+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(326190), waiting 59.990161952s before retrying
2023-03-29T14:47:17.006+0900    WARN    sectors pipeline/fsm.go:799 sector 326248 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326248},"ProofType":8},"e4ZFQO51z8ggVz3LH/kCWX8qVaUDm2qI+x/hoM6NI5A=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqc67vn74jjp4sgshce4excjqbiptjdnrohtanvpoxvc43p2kdqily"}}]]]), got ()
2023-03-29T14:47:17.007+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(326248), waiting 59.992732461s before retrying
2023-03-29T14:47:22.024+0900    INFO    sectors pipeline/states_sealing.go:571  scheduling seal proof computation...
2023-03-29T14:47:22.024+0900    INFO    sectors pipeline/states_sealing.go:573  KOMIT 325933 7a56f40e26f5253a658d955e92639479537f3e85c3cb512d83ab8ff294c153bb(2723429); c04af2053f3f54cd32953bd132b28d59584a51dc4733351307bab36bd2e5909c(2724905); [{34359738368 baga6ea4seaqcaezwci2ryjn5jym4q5t2bhrkpkh2jbaar5spnoxr6tqi4zewyki}]; r:bagboea4b5abca3m4tz6655qppg6seupgiva6uanngwp7zd5tixmwp5i6m23lvrtn; d:baga6ea4seaqcaezwci2ryjn5jym4q5t2bhrkpkh2jbaar5spnoxr6tqi4zewyki
2023-03-29T14:47:22.025+0900    INFO    sectors pipeline/states_failed.go:30    ComputeProofFailed(325933), waiting 59.974355867s before retrying
2023-03-29T14:47:27.013+0900    WARN    sectors pipeline/fsm.go:799 sector 325572 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325572},"ProofType":8},"/SaZ/KkIdRPjYwOkAKuHuHyUp96bcJtjR01qKTSa3vc=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqhrydfzpnxugiv5of5dd6ylllp6kmf3lw5h7vueo7drz5frnz6kji"}}]]]), got ()
2023-03-29T14:47:27.015+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325572), waiting 59.984434359s before retrying
2023-03-29T14:47:29.000+0900    WARN    sectors pipeline/fsm.go:799 sector 326188 got error event sealing.SectorSealPreCommit1Failed: consecutive compute fails
2023-03-29T14:47:29.002+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(326188), waiting 59.997433153s before retrying
2023-03-29T14:47:29.004+0900    WARN    sectors pipeline/fsm.go:799 sector 326431 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326431},"ProofType":8},"ZLZDGrNiFrzaV51oJbYWEl3mgjy+E6JnQVNkvA8TQ4Q=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqjanafdzq5dgncftaczypvitfpfrdyg25nku5xtql56efvq6756ba"}}]]]), got ()
2023-03-29T14:47:29.006+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(326431), waiting 59.993972217s before retrying
2023-03-29T14:47:29.009+0900    WARN    sectors pipeline/fsm.go:799 sector 325682 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325682},"ProofType":8},"rQUxKgBvw2DirCnRFgY8Q1Cj87WgODDVU5kurPFfVjs=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqo34t2pne67riyxygptekaxkwiukhkvfgxo4e2m3ofy6khzdenukq"}}]]]), got ()
2023-03-29T14:47:29.011+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325682), waiting 59.988613003s before retrying
2023-03-29T14:47:36.003+0900    WARN    sectors pipeline/fsm.go:799 sector 326412 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326412},"ProofType":8},"SasEpJ5hrKQromSPyslCOCJvyjhG9aTCgwkEZ1iiLY4=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqmqewykpi6kpcbxyodo474rfghjxkprbn6zv3by3agb3isdudmkdi"}}]]]), got ()
2023-03-29T14:47:36.004+0900    WARN    sectors pipeline/fsm.go:799 sector 326419 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326419},"ProofType":8},"Ic2uB56knTbkL9FAekHMIFzAxxAw5qYCTdZdhy6ncVE=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqcsxacgnuwu5ydq4vv4ebza4ffj7dphf2y6xy4xxjryejkcgrmkda"}}]]]), got ()
2023-03-29T14:47:36.004+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(326412), waiting 59.995356581s before retrying
2023-03-29T14:47:36.005+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(326419), waiting 59.994806411s before retrying
2023-03-29T14:47:36.006+0900    WARN    sectors pipeline/fsm.go:799 sector 325989 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325989},"ProofType":8},"cDCNyshjkQZdVi0ujfyT86YGe9cBNbRwGRcPdS+wPWM=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqiqfjgz5df6xhjmbfbgbwz2nirfjomwc4todycgouhfgcxvno5eci"}}]]]), got ()
2023-03-29T14:47:36.008+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325989), waiting 59.991381547s before retrying
2023-03-29T14:47:38.008+0900    WARN    sectors pipeline/fsm.go:799 sector 325044 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325044},"ProofType":8},"3BDy6QSMyuKo/UlE1zeHh9j6Bh07AtkPaM2sya7z3Ck=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqnligaakkuxm2x3ez7cdvgkwbpqh4bpyvttplwie6bb6aqrngt6iq"}}]]]), got ()
2023-03-29T14:47:38.011+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325044), waiting 59.988537095s before retrying
2023-03-29T14:47:39.007+0900    WARN    sectors pipeline/fsm.go:799 sector 326265 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326265},"ProofType":8},"HDpxqyEgT/LvrBqlEoomDWmPe8c0jCrTu2lyqYMQwPQ=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqkcd4hrlhmsj2aruasausi3bf43uzligxekrrxvbgwpwgexydnema"}}]]]), got ()
2023-03-29T14:47:39.008+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(326265), waiting 59.991422116s before retrying
2023-03-29T14:47:39.011+0900    WARN    sectors pipeline/fsm.go:799 sector 325432 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325432},"ProofType":8},"55U2MxORM9qHb3orIKdcZ3X4IKZHvjBkOWNQtZvi1LU=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqgezndmeopjqytlkf25rzmiwwv4gx5lb4sg7jjdxoag4lwkimvgiy"}}]]]), got ()
2023-03-29T14:47:39.016+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325432), waiting 59.983983626s before retrying
2023-03-29T14:47:40.034+0900    INFO    miner   miner/miner.go:478  completed mineOne   {"tookMilliseconds": 15, "forRound": 2725416, "baseEpoch": 2725415, "baseDeltaSeconds": 10, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2821261, "lookbackEpochs": 900, "networkPowerAtLookback": "22416531296104349696", "minerPowerAtLookback": "19488925475143680", "isEligible": true, "isWinner": false, "error": null}
2023-03-29T14:47:43.009+0900    WARN    sectors pipeline/fsm.go:799 sector 325495 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325495},"ProofType":8},"D9lJoOO/wJPOG9CDHhWTOdaC2DSRx6Xt9aKf/LslyCs=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqloila7ojjsrc3anam53ypgutavuguhoik66cqgjhsic66wovc2ma"}}]]]), got ()
2023-03-29T14:47:43.011+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325495), waiting 59.988686518s before retrying
2023-03-29T14:47:45.175+0900    WARN    sectors pipeline/fsm.go:799 sector 326427 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: storage call error 0: cleaning up sealed data: remove from local: finding existing sector {1697248 326427}(t:2) failed: RPC client error: sendRequest failed: Post "http://10.10.35.7:2345/rpc/v0": read tcp 10.10.35.103:47112->10.10.35.7:2345: read: connection reset by peer [name: porep-35-103]: cleaning up sealed data: remove from local: finding existing sector {1697248 326427}(t:2) failed: RPC client error: sendRequest failed: Post "http://10.10.35.7:2345/rpc/v0": read tcp 10.10.35.103:47112->10.10.35.7:2345: read: connection reset by peer
2023-03-29T14:47:45.175+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(326427), waiting 59.824083523s before retrying
2023-03-29T14:47:45.888+0900    INFO    sectors pipeline/states_sealing.go:429  submitting precommit for sector 326266 (deposit: 79102111037133547): 
2023-03-29T14:47:46.286+0900    INFO    sectors pipeline/states_sealing.go:473  Sector precommitted: 326266
2023-03-29T14:47:51.009+0900    WARN    sectors pipeline/fsm.go:799 sector 325871 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325871},"ProofType":8},"c+gAh98oVXKiGw6S/fHbAYF7q9SuiRvDdljqJSKig9M=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqkovovg2tipaofftwxtzylklewavbsxch42gmacuw2ayl6w2ux6ay"}}]]]), got ()
2023-03-29T14:47:51.011+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325871), waiting 59.988097483s before retrying
2023-03-29T14:47:53.006+0900    WARN    sectors pipeline/fsm.go:799 sector 325739 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325739},"ProofType":8},"7mPubSGG6Py2Jkk6/WP6IRTdJxmhW6BqUmuE49VhctQ=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqkfndsdaqmermbpexqslmvrxlfjrsrxo5ftjritcvdncxwaefokmq"}}]]]), got ()
2023-03-29T14:47:53.008+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325739), waiting 59.991336617s before retrying
2023-03-29T14:47:59.726+0900    INFO    sectors pipeline/states_sealing.go:571  scheduling seal proof computation...
2023-03-29T14:47:59.726+0900    INFO    sectors pipeline/states_sealing.go:573  KOMIT 326208 8389cd049a29d71f0c81431998ef3d1e37c52152c541955dcde44cb8fc155b88(2723959); ffab43c1fa8aa41a4bb92442b4960cc725ffcc17c43d63eda1c6ef49aaf5bd52(2725410); [{34359738368 baga6ea4seaqbazty6cu2vfsew3u4xa25npfkbsybaf5mfvocgsgmfb5leptggoq}]; r:bagboea4b5abca57al6ar7dztfh67oslewjeuwsvcojpikuymswdjox52gwrir3bz; d:baga6ea4seaqbazty6cu2vfsew3u4xa25npfkbsybaf5mfvocgsgmfb5leptggoq
2023-03-29T14:47:59.727+0900    INFO    sectors pipeline/states_sealing.go:571  scheduling seal proof computation...
2023-03-29T14:47:59.727+0900    INFO    sectors pipeline/states_sealing.go:573  KOMIT 326129 10b65cc8b6fc694762373f9310653997c60b023ef588dc117c711a667c0027ff(2723786); ffab43c1fa8aa41a4bb92442b4960cc725ffcc17c43d63eda1c6ef49aaf5bd52(2725410); [{34359738368 baga6ea4seaqovryqvjzrofyw754ujfeaq3ouw2xxfnamhm3p3tiy6fmv356d6pi}]; r:bagboea4b5abcbl6ak3blmvs6er4jqc6z55t55swd5uzobtldne2g3bjxwskpmotq; d:baga6ea4seaqovryqvjzrofyw754ujfeaq3ouw2xxfnamhm3p3tiy6fmv356d6pi
2023-03-29T14:48:04.007+0900    WARN    sectors pipeline/fsm.go:799 sector 325680 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325680},"ProofType":8},"aPQfsubT3uUQF3k3wZEzr8gK7r0X8hVpa4a9EREROf0=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqfd5ml5tztcfcydskrstmjhyn55bby6xb3ya27chssnadtdhfxefy"}}]]]), got ()
2023-03-29T14:48:04.009+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325680), waiting 59.990061112s before retrying
2023-03-29T14:48:07.008+0900    WARN    sectors pipeline/fsm.go:799 sector 326271 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326271},"ProofType":8},"uSVj9leeFVVrj2FwsiVPdeX43wx7Kir/inoqPKDzn+U=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqjhd4qyf3rpbnvpvlpg4iilo5dhbke23k7vw7jvxz2dzc37jld2mi"}}]]]), got ()
2023-03-29T14:48:07.010+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(326271), waiting 59.989069806s before retrying
2023-03-29T14:48:10.018+0900    INFO    miner   miner/miner.go:478  completed mineOne   {"tookMilliseconds": 12, "forRound": 2725417, "baseEpoch": 2725416, "baseDeltaSeconds": 10, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2821262, "lookbackEpochs": 900, "networkPowerAtLookback": "22416462473548398592", "minerPowerAtLookback": "19488925475143680", "isEligible": true, "isWinner": false, "error": null}
2023-03-29T14:48:12.491+0900    INFO    wdpost  wdpost/wdpost_run.go:412    computing window post   {"cycle": "2023-03-29T14:36:00.320+0900", "batch": 0, "elapsed": 263.640970975, "skip": 1, "err": "some sectors (1) were skipped", "errVerbose": "some sectors (1) were skipped:\n    github.com/filecoin-project/lotus/storage/sealer.(*Manager).generateWindowPoSt\n        /home/downloads/lotus/storage/sealer/manager_post.go:205"}
2023-03-29T14:48:12.491+0900    ERROR   wdpost  wdpost/wdpost_run.go:414    error generating window post: some sectors (1) were skipped {"cycle": "2023-03-29T14:36:00.320+0900"}
2023-03-29T14:48:12.499+0900    WARN    wdpost  wdpost/wdpost_run.go:479    generate window post skipped sectors    {"cycle": "2023-03-29T14:36:00.320+0900", "sectors": [{"Miner":1697248,"Number":312269}], "error": "some sectors (1) were skipped", "errorVerbose": "some sectors (1) were skipped:\n    github.com/filecoin-project/lotus/storage/sealer.(*Manager).generateWindowPoSt\n        /home/downloads/lotus/storage/sealer/manager_post.go:205", "try": 1}
2023-03-29T14:48:12.717+0900    INFO    wdpost  wdpost/wdpost_run.go:397    running window post {"cycle": "2023-03-29T14:36:00.320+0900", "chain-random": "wdH/AUpdKxwON8Xd9YHo++vflKjszu4Qm7OzO9kCTaM=", "deadline": {"CurrentEpoch":2725392,"PeriodStart":2723731,"Index":28,"Open":2725411,"Close":2725471,"Challenge":2725391,"FaultCutoff":2725341,"WPoStPeriodDeadlines":48,"WPoStProvingPeriod":2880,"WPoStChallengeWindow":60,"WPoStChallengeLookback":20,"FaultDeclarationCutoff":70}, "height": "2725392", "skipped": 11}
2023-03-29T14:48:13.008+0900    WARN    sectors pipeline/fsm.go:799 sector 325994 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325994},"ProofType":8},"ccJjop/rEPT9trRMBsUpaqRjSRvWNmcIC6Y2IKe5wpQ=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqglwaz47wfphh6swm53fk6w6235a66aokaudfiac3jng4aza2toea"}}]]]), got ()
2023-03-29T14:48:13.034+0900    INFO    sectors pipeline/states_failed.go:30    SealPreCommit1Failed(325994), waiting 59.965358967s before retrying

filecoin-project / lotus