Gapfill error: "load state tree: failed to load state tree: failed to load hamt node: blockstore: block not found"

[placer14]

Describe the bug:

load state tree:
    github.com/filecoin-project/lily/lens/util.MakeGetActorCodeFunc
        /go/src/github.com/filecoin-project/lily/lens/util/repo.go:247
  - failed to load state tree:
    github.com/filecoin-project/lotus/chain/state.LoadStateTree
        /go/pkg/mod/github.com/filecoin-project/lotus@v1.11.0/chain/state/statetree.go:276
  - failed to load hamt node:
    github.com/filecoin-project/specs-actors/actors/util/adt.AsMap
        /go/pkg/mod/github.com/filecoin-project/specs-actors@v0.9.14/actors/util/adt/map.go:41
  - blockstore: block not found

This also appeared out of the chaineconomics task producing this error (without the detailed stacktrace... these might be different expressions of the same error):

{"level":"error","ts":"2021-09-24T03:58:00.177Z","logger":"lily/chain","caller":"chain/indexer.go:378","msg":"task returned with error","current":1131140,"next":1131141,"task":"chaineconomics","error":"get circulating supply: load state tree: failed to load state tree: failed to load hamt node: blockstore: block not found"}

Steps to Reproduce:

This was executed on our internal mainnet analysis DB over all heights in the DB with a node which was intantiated from https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car on Sept 22 (so the state would not necessarily cover all heights covered in the gapfill).

The job report from the fill which produced this error:

        {
                "ID": 3,
                "Name": "initial-fill-03-806642-1133772",
                "Type": "Fill",
                "Error": "",
                "Tasks": [
                        "actorstatesmultisig",
                        "actorstatespower",
                        "actorstatesraw",
                        "actorstatesreward",
                        "actorstatesverifreg",
                        "blocks",
                        "chaineconomics",
                        "consensus",
                        "implicitmessage",
                        "messages",
                        "msapprovals",
                        "actorstatesinit",
                        "actorstatesmarket"
                ],
                "Running": false,
                "RestartOnFailure": false,
                "RestartOnCompletion": false,
                "RestartDelay": 0,
                "Params": {
                        "maxHeight": "1133772",
                        "minHeight": "806642",
                        "storage": "db"
                },
                "StartedAt": "2021-09-24T03:55:14.323816903Z",
                "EndedAt": "2021-09-24T04:00:22.414538928Z"
        }

Unsure what else would be helpful here, but can add as needed.

Lily Version: v0.8.1

[iand]

Pretty sure this is because you went back too far on the gapfill and the node doesn't have receipts for the height specified (the lotus exports don't contain them)

[placer14]

Okay, let's close this and reopen if we start seeing this in the daily jobs.