"Unknown migration in database" when upgrading client version

jakobilobi commented 7 months ago

Hello zkEVM dev community!

I'm with a node provider (Dwellir) who are running several zkEVM nodes. Trying to upgrade the zkevm-node client from version v0.4.1 has not been successful for us and I'd like to inquire if there is a known solution to the issue. I did some searching in the topics here and on the Discord (Polygon, zkEVM channel) but didn't find anything related so far.

Appreciating any help!

System information

zkEVM Node version: v0.4.2 (upgraded from v0.4.1, prover v3.0.2) OS & Version: Linux Networks: Mainnet, Testnet, Custom (Astar zkEVM)

Expected behaviour

Upgrading zkevm-node client version goes smoothly.

Actual behaviour

When the client software starts up again, it fails in creating a migration plan:

INFO        cmd/run.go:52        Starting application        {"pid": 1784118, "version": "0.4.2"}
INFO        cmd/run.go:273        running migrations for zkevm-state-db        {"pid": 1784118, "version": "0.4.2"}
INFO        db/db.go:47        running migrations up        {"pid": 1784118, "version": "0.4.2"}
FATAL        cmd/run.go:276        Unable to create migration plan because of 0001.sql: unknown migration in database

This was observed for the 3 different networks mentioned above.

Steps to reproduce the behaviour

For us, just try to do the upgrade again. I guess we've missed something in our setup but we never had this issue before so I don't know what that might be.

Backtrace

2024-01-02T08:13:05.241Z        FATAL        cmd/run.go:276        Unable to create migration plan because of 0001.sql: unknown migration in database
/home/runner/work/zkevm-node/zkevm-node/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()
/home/runner/work/zkevm-node/zkevm-node/log/log.go:223 github.com/0xPolygonHermez/zkevm-node/log.Fatal()
/home/runner/work/zkevm-node/zkevm-node/cmd/run.go:276 main.runMigrations()
/home/runner/work/zkevm-node/zkevm-node/cmd/run.go:258 main.runStateMigrations()
/home/runner/work/zkevm-node/zkevm-node/cmd/run.go:66 main.start()
/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:274 github.com/urfave/cli/v2.(*Command).Run()
/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267 github.com/urfave/cli/v2.(*Command).Run()
/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:332 github.com/urfave/cli/v2.(*App).RunContext()
/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:309 github.com/urfave/cli/v2.(*App).Run()
/home/runner/work/zkevm-node/zkevm-node/cmd/main.go:191 main.main()
/opt/hostedtoolcache/go/1.21.4/x64/src/runtime/proc.go:267 runtime.main()
        {"pid": 1784118, "version": "0.4.2"}
main.runMigrations
        /home/runner/work/zkevm-node/zkevm-node/cmd/run.go:276
main.runStateMigrations
        /home/runner/work/zkevm-node/zkevm-node/cmd/run.go:258
main.start
        /home/runner/work/zkevm-node/zkevm-node/cmd/run.go:66
github.com/urfave/cli/v2.(*Command).Run
        /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:274
github.com/urfave/cli/v2.(*Command).Run
        /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:267
github.com/urfave/cli/v2.(*App).RunContext
        /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:332
github.com/urfave/cli/v2.(*App).Run
        /home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:309
main.main
        /home/runner/work/zkevm-node/zkevm-node/cmd/main.go:191
runtime.main
        /opt/hostedtoolcache/go/1.21.4/x64/src/runtime/proc.go:267

obynonwane commented 7 months ago

@jakobilobi below is the instructions to upgrade your Node version: v0.4.1, Prover version: v3.0.2 Bridge version: v0.3.1. There are some config changes as indicated in the instructions and confirm that you even have this change reflected in your setup.

v0.4.2 is a patch release and do not require any config change and may not require any specific action from partners.

jakobilobi commented 7 months ago

Hello @obynonwane

Thanks for taking the time to respond. The situation I'm describing is not us upgrading to v0.4.1, but from v0.4.1 to v0.4.2. The config changes are already in place since the last upgrade and the nodes have worked well, so that shouldn't be an issue.

What other reasons could there be for this error?

What does the message Unable to create migration plan because of 0001.sql: unknown migration in database mean to you?

jakobilobi commented 6 months ago

I can add to the case that this comes from the synchronizer component.

Is there some way to reset the migrations manually, and perhaps get around it that way?

Would really appreciate some help!

0xPolygonHermez / zkevm-node