Metrics collecting migration crashes

turbolent commented 1 week ago

Current Behavior

When running the migration at v0.35.15-crescendo-preview.28-atree-inlining on TN snapshot devnet50-execution-snapshot-for-migration-12-jun-19 with --report-metrics, the metrics_collecting_migration/ MetricsCollectingMigration crashes.

(See https://www.notion.so/flowfoundation/TN-state-migration-Atree-inlining-C1-0-Jun-19-preview-28-32cbc865894a4a3586770c818b162132 for details about state)

21:20:14 INF migration 8/9 is underway migration=metrics_collecting_migration
21:20:14 INF created account migrations inner_migrations=1 nWorker=128
panic: internal error: unreachable

goroutine 1 [running]:
github.com/onflow/flow-go/cmd/util/ledger/migrations.(*MetricsCollectingMigration).InitMigration(0xc000ca20d0, {{0x197a268, 0xc000989a00}, 0xff, {0x0, 0x0}, {0x1058e363c00, 0x14, 0x1f4}, {0xc000989a10, ...}, ...}, ...)
        /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/migration_metrics_collector.go:83 +0x605
github.com/onflow/flow-go/cmd/util/ledger/migrations.MigrateByAccount({{0x197a268, 0xc000989a00}, 0xff, {0x0, 0x0}, {0xc00035e200, 0x1, 0x1f4}, {0xc000989a10, 0x1, ...}, ...}, ...)
        /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:85 +0x3ad
github.com/onflow/flow-go/cmd/util/ledger/migrations.NewCadence1ValueMigrations.NewAccountBasedMigration.func6(0xfac29c6a20?)
        /Users/bastian/Documents/work/flow-go/cmd/util/ledger/migrations/account_based_migration.go:52 +0x65
github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.extractExecutionStateFromPayloads.newMigration.func1({0xc000d48000?, 0x70c2096, 0xff?})
        /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:364 +0x417
github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.extractExecutionStateFromPayloads({{0x197a268, 0xc000989a00}, 0xff, {0x0, 0x0}, {0xc00035e200, 0x1, 0x1f4}, {0xc000989a10, 0x1, ...}, ...}, ...)
        /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/execution_state_extract.go:243 +0x25f
github.com/onflow/flow-go/cmd/util/cmd/execution-state-extract.run(0xc000472900?, {0x14ebeb9?, 0x4?, 0x14ebebd?})
        /Users/bastian/Documents/work/flow-go/cmd/util/cmd/execution-state-extract/cmd.go:411 +0x148d
github.com/spf13/cobra.(*Command).execute(0x3f590c0, {0xc000460900, 0x11, 0x12})
        /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987 +0xab1
github.com/spf13/cobra.(*Command).ExecuteC(0x3f562c0)
        /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/bastian/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/onflow/flow-go/cmd/util/cmd.Execute()
        /Users/bastian/Documents/work/flow-go/cmd/util/cmd/root.go:63 +0x1a
main.main()
        /Users/bastian/Documents/work/flow-go/cmd/util/main.go:8 +0xf

Expected Behavior

Migration succeeds

Steps To Reproduce

See Notion document

Environment

- Cadence version:
- Network:

turbolent commented 1 week ago

cc @SupunS

turbolent commented 1 week ago

Opened https://github.com/onflow/flow-go/pull/6140 to get the code of all checked contracts / programs, which then can be used to locally run the metrics collecting migration and see which program code is running into the unreachable case.

turbolent commented 4 days ago

This is caused by bad state which is now encountered due to #6122: Some contract names registers contain names of contracts which are actually empty. For example, 0x4184b8bdf78db9eb.FungibleToken is empty.

This is likely caused by insufficient checks for contracts code, before they were added in '22.

We need to filter out effectively empty contracts in the migration. Ideally we would also clean up the state.

cc @onflow/flow-cadence-execution

onflow / cadence