(SnapDeals) Deals stays in the same state even if we `abortUpgrade`

Reiers commented 2 years ago

Checklist

[X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
[X] This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
[X] This is not a new feature request. If it is, please file a feature request instead.
[X] This is not an enhancement request. If it is, please file a improvement suggestion instead.
[X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
[X] I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
[X] I did not make any code changes to lotus.

Lotus component

[ ] lotus daemon - chain sync
[ ] lotus miner - mining and block production
[ ] lotus miner/worker - sealing
[ ] lotus miner - proving(WindowPoSt)
[X] lotus miner/market - storage deal
[ ] lotus miner/market - retrieval deal
[ ] lotus miner/market - data transfer
[ ] lotus client
[ ] lotus JSON-RPC API
[ ] lotus message management (mpool)
[X] Other

Lotus Version

Daemon:  1.14.0-rc3+mainnet+git.065a62af2+api1.5.0
Local: lotus version 1.14.0-rc3+mainnet+git.065a62af2

Describe the Bug

The deal state does not change when lotus-miner sectors update-state --really-do-it <sector> AbortUpgrade They stay in AwaitingPreCommit, or Sealing.

Logging Information

lotus-miner storage-deals list
ProposalCid  DealId  State                         Client                                     Size   Price  Duration
...7qiuak4y  0       StorageDealError              t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  2MiB   0 FIL  603656
...opmpcyjq  2890    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  2MiB   0 FIL  603647
...6claddqq  2891    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  4GiB   0 FIL  603627
...2yuncnaa  2892    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  16GiB  0 FIL  603624
...s6tyq55u  2893    StorageDealSealing            t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  32GiB  0 FIL  603618
...po5ylzgu  2894    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  32GiB  0 FIL  605236
...qou4gf3a  2895    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  4GiB   0 FIL  605074
...liao7joq  2905    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  32GiB  0 FIL  603902
...ngdcseza  0       StorageDealError              t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  8GiB   0 FIL  523220
...h6hoockq  2906    StorageDealSealing            t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  8GiB   0 FIL  603813
...tbhxtz2q  2907    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  8GiB   0 FIL  603790
...7xjrdr4m  2908    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  8GiB   0 FIL  603785
...etqmebmm  2909    StorageDealError              t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  32GiB  0 FIL  523140
...sarcbk4e  2915    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  8GiB   0 FIL  602793
...xvz5qgme  0       StorageDealError              t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  2MiB   0 FIL  602989
...xvpnvezm  2918    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  2MiB   0 FIL  602986
...klwfcumm  2919    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  2MiB   0 FIL  602982
...66vsoxly  2920    StorageDealAwaitingPreCommit  t12j6glkr4vetge3iqw5vxwl4zubjsxlgt4c5iiji  2MiB   0 FIL  602975

lotus-miner info (snippet)

Storage Deals: 14, 152 GiB
      AwaitingPreCommit: 12  112 GiB (Verified: 0 0 B)
      Sealing:           2   40 GiB  (Verified: 0 0 B)

Retrieval Deals (complete): 0, 0 B

Repo Steps

Run lotus-miner sectors snap-up <sector>
lotus-miner sealing abort <jobID>
lotus-miner sectors update-state --really-do-it <sector> AbortUpgrade

Expected behavior:

StorageDealError immediately
or even better, the deal rolls over to the next SnapDealsWaitDeals sector - if cant find a SnapDealsWaitDeals, create a WaitDeals sector since the upgrade failed.

arajasek commented 2 years ago

We should address this in 1.14.0 -- fine to just fail the deal for now, if the better version (trying to match the deal with a waiting / new sector) is harder

ZenGround0 commented 2 years ago

Not sure but it looks like the ability to do this on the markets side doesn't quite exist yet as I've been having trouble getting desired behavior when failing regular deal sector commitments. I'm going to focus on reassigning deals from failed sectors because it looks like a similar scope problem and gets us closer to where we want to be.

jennijuju commented 2 years ago

ability to do this on the markets side doesn't quite exist yet as I've been having trouble getting desired behavior when failing regular deal sector commitments.

@nonsense we could use some of your insight here from a client / market implementation (boost) perspective:

Currently, if sealing failed for the sector a deal is in, what's the storage deal status in that case?
if it doesn't exist, what do you think the market side should show the user (client)? what information do you expect from the lotus sealing pipeline and during what stages?

nonsense commented 2 years ago

Currently, if sealing failed for the sector a deal is in, what's the storage deal status in that case?

At the moment Boost is handling this in a similar way as Lotus markets - a deal is sent to the sealer and then removed on Boost side. So Boost, same as Lotus markets, doesn't support retrying the deal on a new sector, as the data is no longer there.

I think we should as a first step we should handle this correctly, i.e. error the deal if the sector fails to seal.

As a second step I suggest we add functionality to retry the deal, which is not easy as keeping the data around on markets side is not ideal - ideally this should happen on the sealer side I think, but I am not sure.

filecoin-project / lotus