Issue with ithacanet and Tzkt.Sync

pea-io commented 2 years ago

Hello,

I'm trying to deploy a private tzkt on ithacanet but i have an error. My setup is:

a private tezos archive node (ithacanet)
a Aurora Postgresql 13.4
tzkt.Sync and tzkt.Api deployed on kubernetes

The pod starts fine and begin to synchronize and then i have this message in the tzkt.Sync logs:

info: Tzkt.Sync.Services.Observer[0]
      Applied 312193 of 548025
info: Tzkt.Sync.Services.Observer[0]
      Applied 312194 of 548025
info: Tzkt.Sync.Services.Observer[0]
      Applied 312195 of 548025
info: Tzkt.Sync.Services.Observer[0]
      Applied 312196 of 548025
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. Serialization exception - Missed required property metadata
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. Serialization exception - Missed required property metadata
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. Serialization exception - Missed required property metadata
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. Serialization exception - Missed required property metadata

My Tezos node runs fine (sync is ok)

$ tezos-client rpc get /monitor/bootstrapped
Warning:

                 This is NOT the Tezos Mainnet.

           Do NOT use your fundraiser keys on this network.

{ "block": "BL88BSbBhs8eb4M2gMYSG5WFMgKn1owsXAW2Mk5UfhwSNXRbrVS",
  "timestamp": "2022-05-18T08:24:55Z" }

~ $ tezos-client bootstrapped
Warning:

                 This is NOT the Tezos Mainnet.

           Do NOT use your fundraiser keys on this network.

Node is bootstrapped.

Nothing in the Aurora logs.

Could you please help me to debug this ? Thanks.

Regards,

pea-io commented 2 years ago

I will force my node with --history=archive and see if it fixes the issue

Groxan commented 2 years ago

Hey! There are two possible reasons:

your node doesn't have context for old blocks (see details here: https://github.com/baking-bad/tzkt/issues/84);
your node doesn't store metadata exceeding configured size limit (this can be configured by the --metadata-size-limit parameter, e.g. you can set it to unlimited).

pea-io commented 2 years ago

Hi,

Thanks for the explanation. I did the following changes:

start my node with --history=archive --metadata-size-limit=unlimited

To bootstrap my node i used the snapshot provided here: https://ithacanet.xtz-shots.io/#archive-tarball-1

But i still have the same issue.

info: Tzkt.Sync.Services.Observer[0]
      Applied 312195 of 553835
info: Tzkt.Sync.Services.Observer[0]
      Applied 312196 of 553836
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. Serialization exception - Missed required property metadata

Querying the rpc node for these blocks seems to work fine:

$ tezos-client rpc get /chains/main/blocks/312195
...
          "signature":
            "sigXc9mpjHpHvap8Cofbwfd52aQxEGZF1Vw96t5tpo5aJpCdNivkpnRsL1b3uz6RXRVVqNW8AT4VZhyzLZ2511pp7Ba4tW1m" } ] ] }

$ tezos-client rpc get /chains/main/blocks/312196
...
          "signature":
            "sigTz1DinqfYKGvAgnspaSpNzwa7RG5pKevwXhzftU4gt6RAkt66yiqrGkGiDpBtdMCbrqc7fSiuhbFy4xYjSPLSQeMVVcQH" } ] ] }

Same for 312197 but for 312198 i have this at the end:
          "signature":
            "sigUnPDmy8dyPyTctFwmButWeb67Tam9PbXQhNWcshnYvEosKaQKKY8ft8oVqiCgSPWFB1me2xgaAsC4TiDEhNvn36WoBgM2",
          "metadata": "too large" } ] ] }

The indexer is blocked on block 312196:

  "chain": "ithacanet",
  "chainId": "NetXnHfVqm9iesp",
  "cycle": 76,
  "level": 312196,
  "hash": "BLkB2BLaNKsfKGDRWobSnQ9fkXiWcSoTcwbbQp1Ldee3qAqBQiF",
  "protocol": "Psithaca2MLRFYargivpo7YvUr7wUDqyxrdhC5CQq78mRvimz6A",
  "nextProtocol": "Psithaca2MLRFYargivpo7YvUr7wUDqyxrdhC5CQq78mRvimz6A",
  "timestamp": "2022-03-30T18:05:10Z",
  "votingEpoch": 15,
  "votingPeriod": 15,
  "knownLevel": 554526,
  "lastSync": "2022-05-19T07:58:36Z",
  "synced": false,
  "quoteLevel": 312196,
  "quoteBtc": 8.119470435103909e-05,
  "quoteEur": 3.469197042437102,
  "quoteUsd": 3.867559542917106,
  "quoteCny": 24.550107710574906,
  "quoteJpy": 471.53827405581313,
  "quoteKrw": 4675.821988379049,
  "quoteEth": 0.0011270242527178162,
  "quoteGbp": 2.9457692690177844

Are the snapshots provided by xtz-shots.io good enough ? Do i need to change snapshots provider ? Thanks.

pea-io commented 2 years ago

Please find in attachment the result of tezos-client rpc get /chains/main/blocks/{level} for level 312196 and 312197.

312197.txt 312196.txt

Groxan commented 2 years ago

See "metadata": "too large" in the response you shared :)

I think the node snapshot you use to restore the node was created with different --metadata-size-limit, so that some historical blocks are still truncated.

There are two ways how to solve it:

Temporarily switch the indexer to public RPC (e.g. https://rpc.tzkt.io/ithacanet), to pass that block.
Restore the indexer DB from the latest snapshot, where that block is already indexed

pea-io commented 2 years ago

Even with --metadata-size-limit=unlimited ? That's what i don't understand...

I have temporary switched to a public RPC node but i would like to fix this with my node. The setup is done like this:

/usr/local/bin/tezos-node config init --network ithacanet --data-dir /var/run/tezos/node/data --rpc-addr 127.0.0.1:8732 --allow-all-rpc 0.0.0.0:8732 --history=archive --metadata-size-limit=unlimited

And then i untar the archive tarball.

pea-io commented 2 years ago

Hello,

Ok i have fixed the issue. I think that i have some weird block in the snapshot. After using a public rpc to pass the block, i have switched back to my node and everything works fine. Thanks for your help :)

Groxan commented 2 years ago

@pea-io Indeed, the snapshot contains truncated blocks, because the node, from which the snapshot was taken, was likely started without --metadata-size-limit=unlimited

baking-bad / tzkt

Issue with ithacanet and Tzkt.Sync #103