ReplicaUpdateFailed - Githubissues

lbj2004032 commented 1 year ago

Checklist

[X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
[X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
[X] I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
[X] I did not make any code changes to lotus.

Lotus component

[ ] lotus daemon - chain sync
[ ] lotus fvm/fevm - Lotus FVM and FEVM interactions
[ ] lotus miner/worker - sealing
[ ] lotus miner - proving(WindowPoSt/WinningPoSt)
[ ] lotus JSON-RPC API
[ ] lotus message management (mpool)
[X] Other

Lotus Version

lotus-miner version 1.20.1+mainnet+git.8a510e68e

Repro Steps

No response

Describe the Bug

ReplicaUpdateFailed

config.toml

[Fees] MaxPreCommitGasFee = "1.3 FIL" MaxCommitGasFee = "5 FIL" MaxTerminateGasFee = "0.5 FIL" MaxWindowPoStGasFee = "5 FIL" MaxPublishDealsFee = "5.55 FIL" MaxMarketBalanceAddFee = "2.8 FIL" [Fees.MaxPreCommitBatchGasFee] Base = "0 FIL" PerSector = "3.6 FIL" [Fees.MaxCommitBatchGasFee] Base = "0 FIL" PerSector = "3.6 FIL"

Logging Information

2023-04-19T00:46:03.397+0800    ERROR   sectors pipeline/states_replica_update.go:199   handleSubmitReplicaUpdate: error sending message: GasEstimateMessageGas error: message execution failed: exit 16, reason: message failed with backtrace:
00: f01715688 (method 27) -- no valid updates (16)
01: f06 (method 9) -- all or nothing call contained failures: Batch successes 0 / 24, Batch failing: [code=18 at idx=0, code=18 at idx=1, code=18 at idx=2, code=18 at idx=3, code=18 at idx=4, code=18 at idx=5, code=18 at idx=6, code=18 at idx=7, code=18 at idx=8, code=18 at idx=9, code=18 at idx=10, code=18 at idx=11, code=18 at idx=12, code=18 at idx=13, code=18 at idx=14, code=18 at idx=15, code=18 at idx=16, code=18 at idx=17, code=18 at idx=18, code=18 at idx=19, code=18 at idx=20, code=18 at idx=21, code=18 at idx=22, code=18 at idx=23] (16)
 (RetCode=16)
2023-04-19T00:46:03.418+0800    INFO    sectors pipeline/states_failed.go:30    ReplicaUpdateFailed(136926), waiting 59.581922564s before retrying

TippyFlitsUK commented 1 year ago

Thank you for the ticket @lbj2004032!! 🙏

lbj2004032 commented 1 year ago

Thank you for the ticket @lbj2004032!! 🙏

fil+ deals stuck at Staged period fil+ deal set days 240 greater then sectors Remaining period I had excuted lotus-miner sectors extend

Workers: Seal(2) WdPoSt(0) WinPoSt(0)

Storage Deals: 1545, 1.946 TiB Active: 71 32.81 GiB (Verified: 71 32.81 GiB) AwaitingPreCommit: 55 25.09 GiB (Verified: 55 25.09 GiB) Staged: 1419 1.889 TiB (Verified: 1419 1.889 TiB)

Retrieval Deals (complete): 6, 177 GiB

...u4tgubhm 32997822 StorageDealAwaitingPreCommit f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 64MiB 0 FIL 520372 ...bij4avkm 32997824 StorageDealStaged f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 512MiB 0 FIL 520372 ...gh542zjq 32997823 StorageDealAwaitingPreCommit f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 32MiB 0 FIL 520371 ...rvpy35je 32997826 StorageDealStaged f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 512MiB 0 FIL 520371 ...tvi6x4le 32997828 StorageDealStaged f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 512MiB 0 FIL 520370 ...fcylt2xy 32997825 StorageDealAwaitingPreCommit f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 64MiB 0 FIL 520370 ...4om72nju 32997827 StorageDealAwaitingPreCommit f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 256MiB 0 FIL 520369 ...jpdrpzu4 33000485 StorageDealStaged f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 1GiB 0 FIL 520369 ...wikylzyi 33000483 StorageDealStaged f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 512MiB 0 FIL 520368 ...upxasm5m 33000484 StorageDealStaged f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 512MiB 0 FIL 520367 ...vq26eu44 33000487 StorageDealStaged f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 1GiB 0 FIL 520367 ...ag662ipm 33000482 StorageDealAwaitingPreCommit f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 64MiB 0 FIL 520366 ...yoldsliu 33000486 StorageDealStaged f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 512MiB 0 FIL 520365 ...zzhjf26a 33001783 StorageDealAwaitingPreCommit f1djeaxezazhlg5334gla2j5jojkna6qnzxg7ovba 128MiB 0 FIL

lbj2004032 commented 1 year ago

Thank you for the ticket @lbj2004032!! 🙏

the sectors now became Proving status

$ lotus-miner sectors status 136926 SectorID: 136926 Status: Proving CIDcommD: baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq CIDcommR: bagboea4b5abcaaw5toiqjwxsp3dfrx2c6dndylyycn4x3fcx3xgzmur3lzxunyqu Ticket: 8b3a94ed0bdbf253b37207ff867bf2306d0392b9b3f4a2c76484140adcdfdd3f TicketH: 1937377 Seed: aa108463d7de30654413875c418d0b3d8db96bd6b464eaa426386b9e78ed98fa SeedH: 1938925 Precommit: bafy2bzaced3ltdwfidob4gdmhyyi7npxu3gcww5p3e2zjwe6d6mu64ztxlsgy Commit: bafy2bzacecmznrkdlllwx4p6aizxojfi7q7cvt3fnrk65izv7biarochcv76w Deals: [0] Retries: 0

lbj2004032 commented 1 year ago

Thank you for the ticket @lbj2004032!! 🙏

ERROR sectors pipeline/states_replica_update.go:275 handleReplicaUpdateWait: failed to wait for message: failed to load message: ipld: could not find bafy2bzacea6xdnodahbhvbtydhvqet4wkuehj6z6asydvihnsjgmvcm7p7wxm

Shekelme commented 1 year ago

The same error is observed on my side, and after 10 hours the error does not correct itself. The repeating pattern on lotus-miner:

2023-08-17T19:29:56.543+0300    ERROR   sectors pipeline/states_replica_update.go:199   handleSubmitReplicaUpdate: error sending message: GasEstimateMessageGas error: message execution failed: exit 16, reason: message failed with backtrace:
00: f01222595 (method 27) -- no valid updates (16)
01: f06 (method 9) -- all or nothing call contained failures: Batch successes 0 / 1, Batch failing: [code=18 at idx=0] (16)
 (RetCode=16)
2023-08-17T19:29:56.546+0300    INFO    sectors pipeline/states_failed.go:30    ReplicaUpdateFailed(20529), waiting 59.453572251s before retrying

On lotus daemon:

2023-08-17T19:29:56.543+0300^IWARN^Irpc^Igo-jsonrpc@v0.3.1/handler.go:444^Ierror in RPC call to 'Filecoin.MpoolPushMessage': GasEstimateMessageGas error:$
    github.com/filecoin-project/lotus/node/impl/full.(*MpoolAPI).MpoolPushMessage$
        /home/admfc/lotus/node/impl/full/mpool.go:196$
  - message execution failed: exit 16, reason: message failed with backtrace:$
    00: f01222595 (method 27) -- no valid updates (16)$
    01: f06 (method 9) -- all or nothing call contained failures: Batch successes 0 / 1, Batch failing: [code=18 at idx=0] (16)$
     (RetCode=16):$
    github.com/filecoin-project/lotus/node/impl/full.gasEstimateGasLimit$
        /home/admfc/lotus/node/impl/full/gas.go:327$

Version: lotus-miner version 1.23.3+mainnet+git.7bb1f98ac Fees:

[Fees]
  # type: types.FIL
  # env var: LOTUS_FEES_MAXPRECOMMITGASFEE
  MaxPreCommitGasFee = "0.2 FIL"

  # type: types.FIL
  # env var: LOTUS_FEES_MAXCOMMITGASFEE
  MaxCommitGasFee = "0.3 FIL"

  # type: types.FIL
  # env var: LOTUS_FEES_MAXTERMINATEGASFEE
  MaxTerminateGasFee = "0.8 FIL"

  # WindowPoSt is a high-value operation, so the default fee should be high.
  #
  # type: types.FIL
  # env var: LOTUS_FEES_MAXWINDOWPOSTGASFEE
  MaxWindowPoStGasFee = "10 FIL"

  # type: types.FIL
  # env var: LOTUS_FEES_MAXPUBLISHDEALSFEE
  MaxPublishDealsFee = "1 FIL"

  # type: types.FIL
  # env var: LOTUS_FEES_MAXMARKETBALANCEADDFEE
  MaxMarketBalanceAddFee = "0.03 FIL"

  [Fees.MaxPreCommitBatchGasFee]
    # type: types.FIL
    # env var: LOTUS_FEES_MAXPRECOMMITBATCHGASFEE_BASE
    Base = "0.03 FIL"

    # type: types.FIL
    # env var: LOTUS_FEES_MAXPRECOMMITBATCHGASFEE_PERSECTOR
    PerSector = "0.03 FIL"

  [Fees.MaxCommitBatchGasFee]
    # type: types.FIL
    # env var: LOTUS_FEES_MAXCOMMITBATCHGASFEE_BASE
    Base = "0.07 FIL"

    # type: types.FIL
    # env var: LOTUS_FEES_MAXCOMMITBATCHGASFEE_PERSECTOR
    PerSector = "0.07 FIL"

lotus daemon:

[Fees]
  # type: types.FIL
  # env var: LOTUS_FEES_DEFAULTMAXFEE
  DefaultMaxFee = "0.9 FIL"

Shekelme commented 1 year ago

I have extended the sectors for snap deals by 89 days more than deal end epoch.

While the sector's data is

Sector On Chain Info
SealProof:              9
Activation:             1586313
Expiration:             3912180
DealWeight:             0
VerifiedDealWeight:             0
InitialPledge:          0.311970824380596882 FIL

Expiration Info
OnTime:         3913746
Early:          0

So the "OnTime" expiration epoch 3913746 is less than 90 days from deal end epoch...

filecoin-project / lotus

ReplicaUpdateFailed #10688

Checklist

Lotus component

Lotus Version

Repro Steps

Describe the Bug

Logging Information