Failed to apply updates. Missed required property metadata

Hi! First of all thanks for this project; it's great!

I'm trying to use the docker builds with compose. I have tried 1.6.4, 1.7.0, and the latest all with the same issue. I've wiped the data volume each time.

tkzt_sync keeps responding with this general error:

tzkt_sync.1.p3m9pxjvwbib@rimmer    | fail: Tzkt.Sync.Services.Observer[0]
tzkt_sync.1.p3m9pxjvwbib@rimmer    |       Failed to apply updates. Serialization exception - Missed required property metadata

I'm not sure what it is choking on. Perhaps it is because I'm using the newest tezos-node with the updated storage format?

tezos-node --version
6bc29438 (2022-01-11 16:35:24 +0100) (11.1)

docker --version
Docker version 20.10.6, build 370c289

I don't think it matters since I'm using docker but I'm running this on:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

Linux  5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Let me know if you need any more information, thank you.

Hi! That error means that responses from the node RPC (in particular, /chains/main/blocks/{level}) don't contain necessary data (e.g. metadata fields). You are not the first one who asked this question, so I will write a detailed explanation and pin this issue :)

Why blocks may not contain metadata fields

Roughly speaking, there are two types of storage in the node:

block storage - contains all blocks, that consists of a block header + list of operations;
context storage - contains additional data computed for each block, like balance updates, bigmap diffs, baking rights, etc.

So, when you do RPC request /chains/main/blocks/{level} the node looks for that block in the block storage and then looks for the context for that block in the context storage. If there is no context for that block, the node returns only raw block data, which is not enought for the indexer.

Why there may be no context for some blocks

When the node is working and receiving new blocks, it saves the blocks and also computes and saves context for those blocks. For example, if you check the context for the head block (/chains/main/blocks/head/metadata) it will always be available.

However, it may be missed for old blocks due to two possible reasons:

you are using the "rolling" mode, which prunes the context older than 5 cycles (~15 days);
you have just imported a fresh node snapshot.

Importing fresh node snapshots is the most often reason, actually. This happens because snapshots usually don't contain context data, but only blocks data, so when you import a snapshot you import just the block storage, but the context storage is empty for all blocks from the genesis to the last block in the snapshot.

Why TzKT faces this issue

Let's say you have an indexer that stopped syncing at the block 1000 and then you import a fresh node snapshot which is on the block 1010. When you start the indexer, it tries to fetch the block 1001, but your node doesn't have context for it (due to snapshot), and actually for all blocks older than 1010. So the indexer just can't receive necessary data, and this is what you have faced.

How to solve it

The solution is pretty simple:

Import the latest TzKT snapshot (e.g. https://tzkt.fra1.digitaloceanspaces.com/snapshots/tzkt_v1.7_mainnet.backup).
Import the node snapshot (e.g. https://snapshots-tezos.giganode.io) which is older than TzKT's one.

You can check at which block the indexer is by calling API /v1/head, but normally, TzKT exports snapshots daily, so it should be ok to just use node snapshots that are 2-3 days old. This way you will have context for recent blocks, needed to make TzKT catch up with the node.

baking-bad / tzkt