We do not yet have reproduction steps, but here is what we know:
Has only been observed on testnet
Happens roughly at the same time across nodes that are run by the foundation
These nodes regularly export snapshots for network backup
It is unclear whether the error is coming from the read-write instance or the read-only instance
Running irmin 3.7.2
When this occurred earlier this year, we guessed it might be related to GC and read-only instances not properly clearing the LRU. This was fixed in https://github.com/mirage/irmin/pull/2200.
We think there could be a bug in reading data that hasn't been flushed to disk since the append-only file does not allow reading beyond the last flushed offset, but this should never happen given higher level code in pack_store prevents this with the staging hash table (although this could use some inspection). Having a control file for the node at the time of this error would help us disprove this idea since we could compare the last written offset in the control file to the offset in the key that is logged.
Tezos is experiencing a periodic issue on their
testnet
(has not been observed onmainnet
):We do not yet have reproduction steps, but here is what we know:
testnet
When this occurred earlier this year, we guessed it might be related to GC and read-only instances not properly clearing the LRU. This was fixed in https://github.com/mirage/irmin/pull/2200.
We think there could be a bug in reading data that hasn't been flushed to disk since the append-only file does not allow reading beyond the last flushed offset, but this should never happen given higher level code in
pack_store
prevents this with the staging hash table (although this could use some inspection). Having a control file for the node at the time of this error would help us disprove this idea since we could compare the last written offset in the control file to the offset in the key that is logged.