Malformed HAMT in migration

[Stebalien]

When running the state migration on mainnet, I got the following log message immediately after the migration completed:

2021-01-28T01:21:43.973Z        WARN       vm      vm/runtime.go:332       Abortf: failed to load cron events: failed to load hamt node: HAMT node was malformed
2021-01-28T01:21:43.973Z        WARN       vm      vm/runtime.go:145       VM.Call failure: failed to load cron events: failed to load hamt node: HAMT node was malformed (RetCode=20):
    github.com/filecoin-project/specs-actors/v3/actors/builtin.RequireNoErr
        /var/lib/lotus/go/pkg/mod/github.com/filecoin-project/specs-actors/v3@v3.0.1-0.20210127195111-89c94ebeb753/actors/builtin/shared.go:75
2021-01-28T01:21:43.973Z        WARN       vm      vm/runtime.go:376       vmctx send failed: to: f04, method: 5: ret: [], err: failed to load cron events: failed to load hamt node: HAMT node was malformed (RetCode=20)

Now, I was running this long before the actual upgrade epoch so I do expect some errors. But it's worrying that we'd be able to reach an error like this.

I've attached a CAR of the power actor, rooted at bafy2bzaceaygqfrjkpyrtmzywfijshqrcdqroeu7lxeqor3qtpok2j7kfmwhy. Unfortunately, I had to re-compute this state root (the migration doesn't log the final state root) so it might not even contain the error.

I'll investigate this more tomorrow, but I'm posting it here in case someone wants to take a look.

archive.zip

[Stebalien]

We have confirmed that this is a bug, fix in #1372.

package main

import (
    "context"
    "os"

    "github.com/filecoin-project/go-hamt-ipld/v3"
    "github.com/filecoin-project/lotus/lib/blockstore"
    "github.com/filecoin-project/specs-actors/v3/actors/util/adt"
    "github.com/ipfs/go-cid"
    "github.com/ipld/go-car"
)

func check(err error) {
    if err != nil {
        panic(err)
    }
}

func main() {
    fi, err := os.Open("/path/to/car")
    check(err)
    bs := make(blockstore.MemStore)
    _, err = car.LoadCar(bs, fi)
    check(err)
    root, err := cid.Decode("bafy2bzacedbtihnsprdvwkocs4vfawiav5rmbvgeo7sdzyuiktzqo755qdwag")
    check(err)
    store := adt.WrapBlockStore(context.TODO(), bs)
    _, err = hamt.LoadNode(context.TODO(), store, root, hamt.UseTreeBitWidth(6))
    check(err)
}

[Stebalien]

This has been fixed.