search

filecoin-project / lily

capturing on-chain state for the filecoin network
Other
49 stars 44 forks source link

Loading Circulating supply sometimes fails during processing #1068

Closed frrist closed 1 year ago

frrist commented 1 year ago

Describe the bug:

Lily tipset-workers occasionally (but on a somewhat frequent basis) returns the following error while loading the circulating supply: {"level":"error","ts":"2022-09-26T21:14:11.923Z","logger":"lily/tasks","caller":"chaineconomics/task.go:44","msg":"error received while extracting chain economics, closing lens","error":"get circulating supply: load state tree: failed to load state tree bafy2bzaceb64jptf2kel7tudt37scucldm5qtmjjx45xbbglfqfydpuzw4tkm: failed to load hamt node: ipld: could not find bafy2bzaceb64jptf2kel7tudt37scucldm5qtmjjx45xbbglfqfydpuzw4tkm"} When a different worker retries the task, it will eventually execute successfully. It is also worth noting that lily calls a lotus API directly to get this result indicating the error is somewhere in lotus, or lily's usage of the specific API method.

Steps to Reproduce:

Hard to repro, but happens in prod fairly frequently

Lily Version: v0.12.0

frrist commented 1 year ago

This issue was caused by works lacking the required state to process the task. This is an issue with the deployment rather than lily itself.