[Sync stuck]One of lotus fvm stucks sometimes

OneilYang commented 9 months ago

Checklist

[X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
[X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
[X] I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
[X] I did not make any code changes to lotus.

Lotus component

[X] lotus daemon - chain sync
[x] lotus fvm/fevm - Lotus FVM and FEVM interactions
[ ] lotus miner/worker - sealing
[ ] lotus miner - proving(WindowPoSt/WinningPoSt)
[ ] lotus JSON-RPC API
[ ] lotus message management (mpool)
[ ] Other

Lotus Version

lotus version 1.25.1

Repro Steps

Start lotus daemon
Check "lotus sync status"
After long running, sometimes it displays: Worker xxx: (first one) Stage: header sync Elapsed: xx Hour yy M ... <-- stuck here very long time

For now, "lotus sync wait" only displays this worker's output, but chain sync seems ok('lotus-miner info' displays sync ok);

if use splitstore, it will check "sync wait" in the compaction, and then lotus will stuck entirely.

Describe the Bug

Start lotus daemon
Check "lotus sync status"
After long running, sometimes it displays: Worker xxx: (first one) Stage: header sync Elapsed: xx Hour yy M ... <-- stuck here very long time

For now, "lotus sync wait" only displays this worker's output, but chain sync seems ok('lotus-miner info' displays sync ok);

if use splitstore, it will check "sync wait" in the compaction, and then lotus will stuck entirely.

Logging Information

Worker xxx: (first one)
        Stage: header sync
        Elapsed: xx Hour yy M ...  <-- stuck here very long time

OneilYang commented 9 months ago

Is there any overtime check in the lotus worker status?

I supposed the overtime check is necessary, it should save application for many reasons including this one.

OneilYang commented 7 months ago

Hi, the stuck "worker" actually is the one thread of lotus daemon, not the lotus-worker process, I suppose the label should be "area/lotus..." or something about lotus. tks

OneilYang commented 7 months ago

and seems I can reproduce it with long time running on many machines, if you want to debug it. please let me know.

OneilYang commented 7 months ago

I found lots of user have this issue, and seems they dont know how it happened: one of fvm stuck --> now sync still works until splistore start --> splitsotre will check sync status, so lotus is over they just feel the splitstore is not stable enough, actually it's the problem about sync issue;

how to resolve it:

find the root cause, but seems it's not so easy; (it's a libp2p network issue? I guess...)
add overtime protect to fvm, it can avoid this issue and all other issues in the future; (for example 1-2 mins)
splitstone check the sync status with other ways; (skip the problem) thanks~

filecoin-project / lotus