Failed to sync blocks with Geth 1.10.8

[tclemos]

Summary of Bug

This bug is still under investigation

What we know so far is that Hermez Node is not synchronizing blocks when the Ethereum Network Node is running Geth 1.10.8

We have faced issues with Goerli and Rinkeby networks while testing the new version of Geth.

Expected Behavior

It was supposed to sync as usual with other Geth versions

Steps to Reproduce

Configure the Hermez Node to sync data from an Ethereum Node running version 1.10.8 and then execute the node to start the sync process.

System information

• Node version: v1.6.0-rc2
• OS: Linux/OSX

Additional Information:

provided by @mfcastellani

$ geth version
Geth
Version: 1.10.8-stable
Git Commit: 26675454bf93bf904be7a43cce6b3f550115ff90
Architecture: amd64
Go Version: go1.16.6
Operating System: linux
GOPATH=/home/ubuntu
GOROOT=go

error:

2021-08-24T18:03:11Z    DEBUG   synchronizer/synchronizer.go:566        Syncing...      {"block": 5377045, "ethLastBlock": {"Num":5377133,"Timestamp":"2021-08-24T18:02:34Z","Hash":"0x9b7f005c3dd1c5ed1a28f264afe95b35bc98ff6abe3eadea584225927ff9e37d"}}
2021-08-24T18:03:11Z    DEBUG   statedb/statedb.go:266  Making StateDB Reset    {"batch": 29398, "type": "synchronizer"}
2021-08-24T18:03:11Z    WARN    node/node.go:809        Synchronizer.Sync: unknown block
/home/ubuntu/src/github.com/hermeznetwork/hermez-node/synchronizer/synchronizer.go:843 github.com/hermeznetwork/hermez-node/synchronizer.(*Synchronizer).rollupSync()
/home/ubuntu/src/github.com/hermeznetwork/hermez-node/synchronizer/synchronizer.go:603 github.com/hermeznetwork/hermez-node/synchronizer.(*Synchronizer).Sync()
/home/ubuntu/src/github.com/hermeznetwork/hermez-node/node/node.go:744 github.com/hermeznetwork/hermez-node/node.(*Node).syncLoopFn()
/home/ubuntu/src/github.com/hermeznetwork/hermez-node/node/node.go:801 github.com/hermeznetwork/hermez-node/node.(*Node).StartSynchronizer.func1()

        {"err": "unknown block"}

Until this moment we didn't test a Geth full resync, we basically upgraded the node from 1.10.6 or 1.10.7 to 1.10.8 and the issue started to happen, getting back to 1.10.6 seems to fix the problem.

We have experienced the same with 3 different machines so far.

[tclemos]

I've found the problem, Geth 1.10.8 is returning the wrong hash for the blocks, then Hermez Node tries to use an invalid block hash to query the events and then gets the error unknown block

Ref block: https://goerli.etherscan.io/block/5259636

package main

import (
    "context"
    "fmt"
    "math/big"

    "github.com/ethereum/go-ethereum/ethclient"
)

const (
    geth_1_10_7_url = ""
    geth_1_10_8_url = ""

    block = 5259636
)

func main() {

    ctx := context.Background()

    client_1_10_7, err := ethclient.Dial(geth_1_10_7_url)
    panicErr(err)

    client_1_10_8, err := ethclient.Dial(geth_1_10_8_url)
    panicErr(err)

    b_1_10_7, err := client_1_10_7.HeaderByNumber(ctx, big.NewInt(block))
    panicErr(err)

    b_1_10_8, err := client_1_10_8.HeaderByNumber(ctx, big.NewInt(block))
    panicErr(err)

    fmt.Printf("Number - b_1_10_7: %v b_1_10_8: %v\n", b_1_10_7.Number, b_1_10_8.Number)
    fmt.Printf("Hash - b_1_10_7: %v b_1_10_8: %v\n", b_1_10_7.Hash(), b_1_10_8.Hash())
    fmt.Printf("Hash Hex - b_1_10_7: %v b_1_10_8: %v\n", b_1_10_7.Hash().Hex(), b_1_10_8.Hash().Hex())

    if b_1_10_7.Number.Cmp(b_1_10_8.Number) != 0 {
        panic("hold on, wait a minute, different block number")
    }

    if b_1_10_7.Hash() != b_1_10_8.Hash() {
        panic("hold on, wait a minute, different block hash")
    }

    if b_1_10_7.Hash().Hex() != b_1_10_8.Hash().Hex() {
        panic("hold on, wait a minute, diffferent block hash hex")
    }
}

func panicErr(err error) {
    if err != nil {
        panic(err)
    }
}

output:

Number - b_1_10_7: 5259636 b_1_10_8: 5259636
Hash - b_1_10_7: 0x3705a71830dab88c25ca3b5765ffba738bea10339d629b4fb225679b5d8be9de b_1_10_8: 0xc7c6eee418abf033c0d1247598dc73ad39e047573fe703c8e995716534725cee
Hash Hex - b_1_10_7: 0x3705a71830dab88c25ca3b5765ffba738bea10339d629b4fb225679b5d8be9de b_1_10_8: 0xc7c6eee418abf033c0d1247598dc73ad39e047573fe703c8e995716534725cee
panic: hold on, wait a minute, different block hash

@jbaylina @jeffprestes @mfcastellani

[tclemos]

I've opened an issue to Geth team: https://github.com/ethereum/go-ethereum/issues/23463

They are already working on a fix reverting the change that introduced the bug: https://github.com/ethereum/go-ethereum/pull/23466