Sync issues on Cardano Node v8.7.3

[YuXiaoCoder]

Ask a question

Hi,

I am running a Cardano node on the mainnet with the following versions:

ghcr.io/intersectmbo/cardano-node:8.7.3
cardanosolutions/ogmios:v6.1.0
postgres:16.2-alpine
cardanofoundation/cardano-graphql-server:8.0.1-mainnet
cardanofoundation/cardano-graphql-background:8.0.1-mainnet
ghcr.io/intersectmbo/cardano-db-sync:13.2.0.2
cardanofoundation/cardano-graphql-hasura:8.0.1

I am using the compatible configuration files located here LINK.

Due to the database upgrade, I reloaded the latest db-sync snapshot and cardano-node snapshot

# cardano-node
wget --no-check-certificate https://downloads.csnapshots.io/mainnet/$(curl -s -k https://downloads.csnapshots.io/mainnet/mainnet-db-snapshot.json | grep -o 'mainnet-db-[0-9]*.tar.lz4')

# db-sync
https://update-cardano-mainnet.iohk.io/cardano-db-sync/13.2/db-sync-snapshot-schema-13.2-block-10208405-x86_64.tgz

After the service starts, everything works fine, but cardano-graphql-server is not populated with Asset data, which prevents the interface from being used

kubectl logs -f ada-2-0 ada-2-cardano-graphql-server --tail 10

{"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-04-25T07:13:11.178Z","v":0}
{"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"CardanoNodeClient","tip":{"slot":122462877,"id":"6ba0e27077bd656e381af575faa0b21f104658f8eae804f192c07bdb60580205"},"msg":"[Object: null prototype] {\n  slot: 122462877,\n  id: '6ba0e27077bd656e381af575faa0b21f104658f8eae804f192c07bdb60580205'\n}","time":"2024-04-25T07:13:11.181Z","v":0}
{"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 100% | Asset: 0%","time":"2024-04-25T07:13:24.157Z","v":0}
{"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-04-25T07:13:24.158Z","v":0}
{"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"CardanoNodeClient","tip":{"slot":122462905,"id":"843ccb435fd90c3cb35ecb4db28f42435d8fb93f198abf7fff082dfde6819a9d"},"msg":"[Object: null prototype] {\n  slot: 122462905,\n  id: '843ccb435fd90c3cb35ecb4db28f42435d8fb93f198abf7fff082dfde6819a9d'\n}","time":"2024-04-25T07:13:24.161Z","v":0}
{"name":"cardano-graphql","hostname":"ada-2-0","pid":365,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-04-25T07:13:29.158Z","v":0}

It would be great if you could help me with the node sync issue.

[YuXiaoCoder]

@Kammerlo Can you help me with this problem, thank you very much!

[Kammerlo]

@YuXiaoCoder I updated the configs so this should be working now. Currently I changed it to rely on the standard config files which are delivered by the ogmios container. You can change this (if you want to run it with different configs) by changing this within the docker-compose.yml Could you try it again?

[YuXiaoCoder]

@Kammerlo Ok, I'll try it and give a reply, thanks!

[YuXiaoCoder]

@Kammerlo There may still be a problem, the Asset is still empty and the progress is zero.

[YuXiaoCoder]

cardano-graphql-hasura startup error: table does not exist

{"detail":{"info":{"objects":[{"definition":{"name":"Cardano","schema":"public"},"name":"table Cardano in source default","reason":"Inconsistent object: no such table/view exists in source: \"Cardano\"","type":"table"},{"definition":{"name":"ProtocolParams","schema":"public"},"name":"table ProtocolParams in source default","reason":"Inconsistent object: no such table/view exists in source: \"ProtocolParams\"","type":"table"},{"definition":{"name":"ReferenceInput","schema":"public"},"name":"table ReferenceInput in source default","reason":"Inconsistent object: no such table/view exists in source: \"ReferenceInput\"","type":"table"},{"definition":{"name":"Block","schema":"public"},"name":"table Block in source default","reason":"Inconsistent object: no such table/view exists in source: \"Block\"","type":"table"},{"definition":{"name":"Transaction","schema":"public"},"name":"table Transaction in source default","reason":"Inconsistent object: no such table/view exists in source: \"Transaction\"","type":"table"},{"definition":{"name":"ActiveStake","schema":"public"},"name":"table ActiveStake in source default","reason":"Inconsistent object: no such table/view exists in source: \"ActiveStake\"","type":"table"},{"definition":{"name":"Delegation","schema":"public"},"name":"table Delegation in source default","reason":"Inconsistent object: no such table/view exists in source: \"Delegation\"","type":"table"},{"definition":{"name":"RedeemerDatum","schema":"public"},"name":"table RedeemerDatum in source default","reason":"Inconsistent object: no such table/view exists in source: \"RedeemerDatum\"","type":"table"},{"definition":{"name":"StakePoolOwner","schema":"public"},"name":"table StakePoolOwner in source default","reason":"Inconsistent object: no such table/view exists in source: \"StakePoolOwner\"","type":"table"},{"definition":{"name":"Epoch","schema":"public"},"name":"table Epoch in source default","reason":"Inconsistent object: no such table/view exists in source: \"Epoch\"","type":"table"},{"definition":{"name":"TransactionInput","schema":"public"},"name":"table TransactionInput in source default","reason":"Inconsistent object: no such table/view exists in source: \"TransactionInput\"","type":"table"},{"definition":{"name":"Utxo","schema":"public"},"name":"table Utxo in source default","reason":"Inconsistent object: no such table/view exists in source: \"Utxo\"","type":"table"},{"definition":{"name":"TokenMint","schema":"public"},"name":"table TokenMint in source default","reason":"Inconsistent object: no such table/view exists in source: \"TokenMint\"","type":"table"},{"definition":{"name":"SlotLeader","schema":"public"},"name":"table SlotLeader in source default","reason":"Inconsistent object: no such table/view exists in source: \"SlotLeader\"","type":"table"},{"definition":{"name":"CollateralInput","schema":"public"},"name":"table CollateralInput in source default","reason":"Inconsistent object: no such table/view exists in source: \"CollateralInput\"","type":"table"},{"definition":{"name":"Reward","schema":"public"},"name":"table Reward in source default","reason":"Inconsistent object: no such table/view exists in source: \"Reward\"","type":"table"},{"definition":{"name":"AdaPots","schema":"public"},"name":"table AdaPots in source default","reason":"Inconsistent object: no such table/view exists in source: \"AdaPots\"","type":"table"},{"definition":{"name":"Withdrawal","schema":"public"},"name":"table Withdrawal in source default","reason":"Inconsistent object: no such table/view exists in source: \"Withdrawal\"","type":"table"},{"definition":{"name":"CollateralOutput","schema":"public"},"name":"table CollateralOutput in source default","reason":"Inconsistent object: no such table/view exists in source: \"CollateralOutput\"","type":"table"},{"definition":{"name":"StakePool","schema":"public"},"name":"table StakePool in source default","reason":"Inconsistent object: no such table/view exists in source: \"StakePool\"","type":"table"},{"definition":{"name":"StakePoolRetirement","schema":"public"},"name":"table StakePoolRetirement in source default","reason":"Inconsistent object: no such table/view exists in source: \"StakePoolRetirement\"","type":"table"},{"definition":{"name":"StakeRegistration","schema":"public"},"name":"table StakeRegistration in source default","reason":"Inconsistent object: no such table/view exists in source: \"StakeRegistration\"","type":"table"},{"definition":{"name":"StakeDeregistration","schema":"public"},"name":"table StakeDeregistration in source default","reason":"Inconsistent object: no such table/view exists in source: \"StakeDeregistration\"","type":"table"},{"definition":{"name":"Script","schema":"public"},"name":"table Script in source default","reason":"Inconsistent object: no such table/view exists in source: \"Script\"","type":"table"},{"definition":{"name":"TokenInOutput","schema":"public"},"name":"table TokenInOutput in source default","reason":"Inconsistent object: no such table/view exists in source: \"TokenInOutput\"","type":"table"},{"definition":{"name":"Datum","schema":"public"},"name":"table Datum in source default","reason":"Inconsistent object: no such table/view exists in source: \"Datum\"","type":"table"},{"definition":{"name":"Redeemer","schema":"public"},"name":"table Redeemer in source default","reason":"Inconsistent object: no such table/view exists in source: \"Redeemer\"","type":"table"},{"definition":{"name":"TransactionOutput","schema":"public"},"name":"table TransactionOutput in source default","reason":"Inconsistent object: no such table/view exists in source: \"TransactionOutput\"","type":"table"},{"definition":{"comment":null,"name":"firstAppearedInBlock","source":"default","table":{"name":"Asset","schema":"public"},"using":{"manual_configuration":{"column_mapping":{"firstAppearedInSlot":"slotNo"},"insertion_order":null,"remote_table":{"name":"Block","schema":"public"}}}},"name":"object_relation firstAppearedInBlock in table Asset in source default","reason":"Inconsistent object: table \"Block\" is not tracked","type":"object_relation"},{"definition":{"comment":null,"name":"stakePool","source":"default","table":{"name":"pool_relay","schema":"public"},"using":{"manual_configuration":{"column_mapping":{"update_id":"update_id"},"insertion_order":null,"remote_table":{"name":"StakePool","schema":"public"}}}},"name":"object_relation stakePool in table pool_relay in source default","reason":"Inconsistent object: table \"StakePool\" is not tracked","type":"object_relation"},{"definition":{"comment":null,"name":"tokenMints","source":"default","table":{"name":"Asset","schema":"public"},"using":{"manual_configuration":{"column_mapping":{"assetName":"assetName","policyId":"policyId"},"insertion_order":null,"remote_table":{"name":"TokenMint","schema":"public"}}}},"name":"array_relation tokenMints in table Asset in source default","reason":"Inconsistent object: table \"TokenMint\" is not tracked","type":"array_relation"},{"definition":{"comment":null,"name":"tokenMints","source":"default","table":{"name":"Asset","schema":"public"},"using":{"manual_configuration":{"column_mapping":{"assetName":"assetName","policyId":"policyId"},"insertion_order":null,"remote_table":{"name":"TokenMint","schema":"public"}}}},"name":"array_relation tokenMints in table Asset in source default","reason":"Inconsistent object: table \"TokenMint\" is not tracked","type":"array_relation"},{"definition":{"comment":null,"name":"transaction","source":"default","table":{"name":"tx_metadata","schema":"public"},"using":{"manual_configuration":{"column_mapping":{"tx_id":"id"},"insertion_order":null,"remote_table":{"name":"Transaction","schema":"public"}}}},"name":"object_relation transaction in table tx_metadata in source default","reason":"Inconsistent object: table \"Transaction\" is not tracked","type":"object_relation"}]},"message":"Inconsistent Metadata!"},"level":"warn","timestamp":"2024-04-27T10:14:10.689+0000","type":"metadata"}

[YuXiaoCoder]

cardano-graphql-background logs:

{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"Db","msg":"Initializing...","time":"2024-04-28T04:51:30.520Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"HasuraBackgroundClient","msg":"Initializing","time":"2024-04-28T04:51:30.525Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"HasuraBackgroundClient","msg":"Initialized","time":"2024-04-28T04:53:09.729Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"MetadataFetchClient","msg":"Initializing","time":"2024-04-28T04:53:09.729Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"MetadataFetchClient","msg":"Initialized","time":"2024-04-28T04:53:10.835Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Initializing","time":"2024-04-28T04:53:10.835Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Initialized","time":"2024-04-28T04:53:10.842Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"Worker","msg":"Starting","time":"2024-04-28T04:53:10.842Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"Worker","msg":"Started","time":"2024-04-28T04:53:10.875Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Starting","time":"2024-04-28T04:53:10.883Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Started","time":"2024-04-28T04:53:10.904Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Rolling back to genesis","time":"2024-04-28T04:53:10.949Z","v":0}
{"name":"background","hostname":"ada-2-0","pid":16,"level":30,"module":"ChainFollower","msg":"Deleted 0 assets","time":"2024-04-28T04:53:10.952Z","v":0}
/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:85
                                    for (const entry of Object.entries(tx.mint.assets)) {
                                                               ^

TypeError: Cannot convert undefined or null to object
    at Function.entries (<anonymous>)
    at ChainFollower.<anonymous> (/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:85:64)
    at Generator.next (<anonymous>)
    at /app/packages/api-cardano-db-hasura/dist/ChainFollower.js:31:71
    at new Promise (<anonymous>)
    at __awaiter (/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:27:12)
    at Object.rollForward (/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:80:62)
    at handler (/app/node_modules/@cardano-ogmios/client/dist/ChainSynchronization/nextBlock.js:23:46)
    at messageHandler (/app/node_modules/@cardano-ogmios/client/dist/ChainSynchronization/Client.js:17:43)
    at asyncWrapper (/app/node_modules/fastq/queue.js:213:12)

Node.js v18.19.1

[Kammerlo]

@YuXiaoCoder Hey, sorry for letting you wait! I'm currently working on this issue. I already found the problem why the chainfollower crashes and the assets were empty. I'm doing everything in this branch: fix/syncing-issues-assets-missing I'll still continue testing and I need to fix the errors within postgres. So if you find anything else, let me know! Sorry again for the delay!

[M2tec]

Thanks for doing this work! Was trying out your branch: fix/syncing-issues-assets-missing. All works fine except for very slow or stuck syncing of graphql. Not getting past Asset: 11%

$ docker logs preprod-server-1 -f {"name":"cardano-graphql","hostname":"ee555697fce1","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 100% | Asset: 11%","time":"2024-05-01T09:29:39.112Z","v":0}.

Let it run all night ~10hours . System: CPU: AMD Ryzen™ 9 7900X3D × 24 Memory: 128 GB. SSD: Corsair MP700 2 TB - SSD m.2 2280 - PCI Express 5.0 x4

Maybe no metadata for preprod, hmm

[YuXiaoCoder]

@Kammerlo Thank you very much for your help, looking forward to your new version

[Kammerlo]

@YuXiaoCoder So this fix solved your issue?

@M2tec I'm currently resyncing to test it. I'll let you know. Did you find any other issues or is it just the 11% related? And did you check that your node + db-sync were at the tip?

[M2tec]

Yeah, works. docker exec -it preprod-cardano-node-ogmios-1 cardano-cli query tip --socket-path /ipc/node.socket --testnet-magic 1

{ "block": 2195749, "epoch": 140, "era": "Babbage", "hash": "000aa984d41418d82cfdc933b0b3cb36c7c5c766717fa920776e03cd8626574e", "slot": 58947531, "slotInEpoch": 109131, "slotsToEpochEnd": 322869, "syncProgress": "100.00" } and

docker logs preprod-cardano-db-sync-1 -f [db-sync-node:Info:64] [2024-05-02 06:19:46.16 UTC] Insert Babbage Block: epoch 140, slot 58947586, block 2195752, hash 35125e5a1fcd9ec3a4c8bfdd273a275c3e51c4344920740aaad3a877c3273e36

I tested some graphQL queries with the hasura console via http://localhost:8091/console

The service is trying to get metadata for the assets. But maybe this does not work for preprod. It puts 77850 assets in a retry state. Logged into the cexplorer database with pgAdmin and did the following query: select * from pgboss.job where name = 'asset-metadata-fetch-initial' and state = 'retry' -- and state is distinct from 'retry' ORDER BY startedon desc;

[cshintov]

It will only crash for a fresh sync? I've already upgraded my mainnet with

ogmios version : v6.2.0

But for preprod did a re-sync from scratch. The mainnet graphql-background doesn't crash because it already had the data for Asset table?

But mainnet Asset is stuck at 83%!

How do I fix the mainnet once your code is merged? Will it be auto-fixed?

[Kammerlo]

@cshintov For mainnet a resync is needed as well, since we needed to upgrade the db and due to some changes of the schema. Did you just upgraded or the errors occur even after a resync?

[cshintov]

For the mainnet I didn't resync. But there were no errors when I did the upgrade. Eventhough the Assets is stuck at 83%

{"name":"cardano-graphql","hostname":"a2db8d7e43f2","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 100% | Asset: 83%","time":"2024-05-02T13:02:43.663Z","v":0}

But for preprod my node db was corrupted so I did resync. But when I started the graphql afterwards the above mentioned error happened.

[cshintov]

For mainnet a resync is needed as well

You mean I have to nuke db-sync-data, node-db and postgres or only postgres for mainnet?

[Kammerlo]

db-sync and postgres. The node should be fine. It's just because the new version of db-sync needs a newer Postgres version and thats why a resync is needed

[cshintov]

Did the latest upgrade for 8.0.1 on preprod. Asset is stuck at 11%!

{"name":"cardano-graphql","hostname":"07816968a372","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 100% | Asset: 11%","time":"2024-05-03T05:45:52.449Z","v":0}

been 8 hours!

Current logs for background:

{"name":"background","hostname":"c29a3a8c82fa","pid":1,"level":30,"module":"HasuraClient","assetId":"2cd858d024a08ec3049bbdc05e27dc894635de72b2b35d09f7cf8874","msg":"Adding metadata to asset","time":"2024-05-03T05:48:23.500Z","v":0}

[Kammerlo]

@cshintov Fixed it in this branch: fix/assets-skipped. Could you check and verify if this solves your issue?

Logs from server: {"name":"cardano-graphql","hostname":"d7e9401ef017","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 85.36% | Asset: 100%","time":"2024-05-03T08:49:43.897Z","v":0}

If this solves your issue I will merge it as soon as possible.

[cshintov]

Will have to build the image. Can you point me towards the Dockerfile?

[Kammerlo]

you can checkout the branch and then start the docker compose with --build at the end. For example NETWORK=preprod METADATA_SERVER_URI="https://metadata.world.dev.cardano.org/metadata" docker compose up -d --build It will build all containers from scratch then

[M2tec]

Tested fix/assets-skipped also.

docker logs preprod-server-1 -f {"name":"cardano-graphql","hostname":"b52c36455241","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 100% | Asset: 100%","time":"2024-05-03T10:37:28.583Z","v":0}

Slight nitpick, sorry Underlying GraphQL query: query getSyncStatus { cardanoDbMeta { initialized syncPercentage } assets_aggregate { aggregate { count } } tokenMints_aggregate(distinct_on: assetId) { aggregate { count } } }

Result mint is off by 1: { "data": { "cardanoDbMeta": { "initialized": true, "syncPercentage": 100 }, "assets_aggregate": { "aggregate": { "count": "712275" } }, "tokenMints_aggregate": { "aggregate": { "count": "712274" } } } }

But "cardanoDbMeta": { "initialized": true}. Have not seen this before. Many thanks!

[Kammerlo]

Perfect!

[Kammerlo]

Closing this issue since the bugfix is now in PreRelease 8.0.2 If you have any other issues, let me know.

[cshintov]

Yup. It worked. Thanks 🙂

I've another issue, but it's related to db-sync. Can't use db sync snapshot with v13.2.0.1 on mainnet. The pg_restore is of version 11 whereas the dump is version 14!

Tried to build a new image but hitting error!

nix-store -l /nix/store/9f4gsmh0bxd1mdhsd34gw4a5vc8fvn1d-docker-layer-cardano-db-sync-base-env.drv
Formatting './image/disk-image.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16
Could not access KVM kernel module: Permission denied
qemu-kvm: failed to initialize kvm: Permission denied
qemu-kvm: falling back to tcg

The user has permission to access kvm, can run qemu directly, but nix build fails!

Will raise an issue there!

[cshintov]

By the way, why there's no 8.0.2-preprod for graphql image? There's 8.0.1-preprod!

It doesn't matter? I can use 8.0.2 image for both mainnet and preprod?

Step 2/16 : FROM cardanofoundation/cardano-graphql:${version}-preprod                                                                                                                                      ║  ║
║  ║ manifest for cardanofoundation/cardano-graphql:8.0.2-preprod not found: manifest unknown: manifest unknown