Frozen block numbers across epochs

[tilacog]

Block numbers reported by the Epoch Subgraph remain the same as epochs progress.

We observed the following behavior in staging:

1. The first SetBlockNumbersForNextEpoch message correctly updates all registered chains.
2. The second message decelerates them to an amount equal to the current block number. The delta is also zero.
3. Subsequent messages have a zero for both acceleration and delta.

Effectively, this is causing the block numbers to stay immutable across epochs.

It is still uncertain whether this bug is happening on the Oracle or Subgraph.

[tilacog]

I'm observing a similar behavior in the local environment of the current main branch in which indexed chains stay at block zero.

I'm unfamiliar with that part of the code, but I tried debugging the Encoder::compress_block_ptrs function. I observed that in the very beginning of this function we are transforming the keys of the block_ptrs argument into default Networks (with zeroed block numbers and hashes).

Here is some debug output I've managed to collect:

// This is the value as it arrives in the function
[crates/encoding/src/lib.rs:181] block_ptrs = {
    "eip155:77": BlockPtr {
        number: 27116446,
        hash: "0xd5df7bb5a5fa35d3541543ad6c5781fe501ca0d893dc0e982db590fd04e3d171",
    },
    "eip155:99": BlockPtr {
        number: 28601784,
        hash: "0x05e7a6c9c3309624280cff9db324a3ed0100874184510579702b7c01c48f78e5",
    },
}

// This is how it exits the first `for` loop (the second expression in the function).
[crates/encoding/src/lib.rs:193] &block_ptrs = {
    "eip155:77": BlockPtr {
        number: 0,
        hash: "0x0000000000000000000000000000000000000000000000000000000000000000",
    },
    "eip155:99": BlockPtr {
        number: 0,
        hash: "0x0000000000000000000000000000000000000000000000000000000000000000",
    },
}

If I understood correctly, we are replacing its keys (Networks) with the ones already present in self.networks, which, in turn, are instantiated using the Default::default() call. That could explain the zeroed Networks I'm looking at.

I'm not sure if this is the expected behavior, so please ignore this report if it is.

The zeroed block_ptrs value is then used to calculate deltas and accelerations through the rest of this function, which results in the creation of compressed messages with acceleration = 0