search

cardano-foundation / cardano-graphql

GraphQL API for Cardano
Apache License 2.0
259 stars 103 forks source link

Sync issues on Cardano Node v8.7.3 #866

Closed saravadeanil closed 4 months ago

saravadeanil commented 4 months ago

Ask a question

Hi,

I am running a Cardano node on the preview with the following versions:

ghcr.io/intersectmbo/cardano-node:8.7.3
cardanosolutions/ogmios:v6.1.0
postgres:14.10-alpine
cardanofoundation/cardano-graphql-server:8.0.1-preview
cardanofoundation/cardano-graphql-background:8.0.1-preview
ghcr.io/intersectmbo/cardano-db-sync:13.2.0.1
cardanofoundation/cardano-graphql-hasura:8.0.1

I am using the compatible configuration files located here LINK.

I started all the components from scratch due to a database upgrade and resync was needed for this version upgrade.

All my Cardano node components are running, but my node is not syncing since 1 day.

cardano-cli query tip --testnet-magic 2 --socket-path /ipc/node.socket
{
    "epoch": 0,
    "era": "Alonzo",
    "slotInEpoch": 0,
    "slotsToEpochEnd": 86400,
    "syncProgress": "0.00"
}

Following are my observations on the setup:

  1. The cardano-node pod seems to have the latest block number 2000941 with a ShutdownPeer message in the logs.
[cardano-:cardano.node.ConnectionManager:Info:620603] [2024-04-18 09:48:52.08 UTC] TrConnectionHandler (ConnectionId {localAddress = 10.0.1.3:43923, remoteAddress = 99.80.240.19:3001}) (TrConnectionHandlerError OutboundError (HeaderError (At (Block {blockPointSlot = SlotNo 0, blockPointHash = 268ae601af8f9214804735910a3301881fbe0eec9936db7d1fb9fc39e93d1e37})) (HeaderProtocolError (HardForkValidationErrFromEra S (S (S (S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (VRFKeyBadNonce (Nonce "81e47a19e6b29b0a65b9591762ce5143ed30d0261e5d24a3201752506b20f15c") (SlotNo 0) (Nonce "816ae1d6589c944bb40011e44fdf0531b6702d3f6e569c62d7ba08a0367cc412") (CertifiedVRF {certifiedOutput = OutputVRF {getOutputVRFBytes = "\213\211\US\141\181\nE\152\206\210\DLE\133\DEL\186\148\169\130\USe\184o\228\181gQiB\186\&1\247\188\147[~\150nUL\RSN\247\200n\173%\136\170\fd\141\150\239\178\171w\183\151C\208\&3\vz\fH"}, certifiedProof = CertPraosVRF "]9\226;\227\181r i\ESC\182^\DC2\133J\237e\147\233\133J\RS\156DL\151k\138\EOTm\223?\STX\247w\200\DC3\147X\236\229'b2l($\236\132\233\SUB3\166lV\180\197\153q\220\229&\147\200C\185Z\ESC\232n\NUL1<\211\218\197|\154e\a"}))]}))))))) TipGenesis (Tip (SlotNo 46777728) 84dc4ab1ad03583813cdb5c414c5086e959d20d64ceed369294ab178b4238be4 (BlockNo 2000941))) ShutdownPeer)
  1. The cardano-graphql-background pod terminates after a certain while with the following error:
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":50,"msg":"Epoch information not yet available. This is expected during the initial chain-sync.","time":"2024-04-18T11:31:08.630Z","v":0}

  1. The Postgres pod has constant error messages. Attaching logs from the Postgres pod on the issue.

  2. The cardano-db-sync-extended pod doesn't seem to be syncing after the initial bootstrap.

I'm attaching logs from all the components on the issue. cardano-logs-v8.7.3.zip

It would be great if you could help me with the node sync issue.

Kammerlo commented 4 months ago

Hi, I'm sorry for the issues! I'm looking into this. You mentioned your node isn't syncing for a day? Did the node sync at all? Because of your attached log I assume not, but I just want to be sure.

saravadeanil commented 4 months ago

You mentioned your node isn't syncing for a day? Did the node sync at all? Because of your attached log I assume not, but I just want to be sure.

@Kammerlo Thanks for your reply!

Yes, your assumption is correct. The node didn't sync at all since from the start. As I started it from scratch.

Kammerlo commented 4 months ago

So I just pushed to master and updated the config files. The node will start to sync now. I will deal with the other issue now, but this will take some effort. I'll notify you, when I'm ready for testing :)

saravadeanil commented 4 months ago

So I just pushed to master and updated the config files. The node will start to sync now. I will deal with the other issue now, but this will take some effort. I'll notify you, when I'm ready for testing :)

Alright, thanks a lot for your help ; ) @Kammerlo

saravadeanil commented 4 months ago

So I just pushed to master and updated the config files. The node will start to sync now. I will deal with the other issue now, but this will take some effort. I'll notify you, when I'm ready for testing :)

Hi @Kammerlo just to inform you, the situation remains unchanged with the new configurations. The node is still not syncing.

Kammerlo commented 4 months ago

What is the node logging? Because when I start it with the configs. It starts syncing? Could you test if it's really an issue with the configuration or anything else by removing the line 40 in the docker-compose.yml? This way the node uses the standard configs provided by cardano-solutions.

saravadeanil commented 4 months ago

@Kammerlo Sure, I can try it once again.

Just to confirm, you mean try removing THIS?

Kammerlo commented 4 months ago

Ah no it's already commented out. It's the mount of the config folder. So just try with the docker-compose.yml from master Sorry for the confusion!

saravadeanil commented 4 months ago

Hi @Kammerlo Thanks a lot!

I ran the docker-compose.yml and it seems the the db-sync and cardano-node-ogmios container is initialized and syncing. Can you please confirm if?

Check Cardano DB sync progress

curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "{ cardanoDbMeta { initialized syncPercentage }}"}' \
  http://localhost:3102/graphql
{"data":{"cardanoDbMeta":{"initialized":false,"syncPercentage":95.54}}}
curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "{ cardano { tip { number slotNo epoch { number } } } }"}' http://localhost:3102/graphql
{"data":{"cardano":{"tip":{"number":106392,"slotNo":"2258381","epoch":null}}}}
cardano-cli query tip --testnet-magic 2 --socket-path /ipc/node.socket
{
    "block": 160640,
    "epoch": 38,
    "era": "Babbage",
    "hash": "85e8644691883defd418f700ba7e3645bb7f93e947ed50b365148e3864506a7d",
    "slot": 3365346,
    "slotInEpoch": 82146,
    "slotsToEpochEnd": 4254,
    "syncProgress": "7.10"
}

container logs

{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"Db","msg":"Initializing...","time":"2024-04-25T16:00:59.324Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":50,"err":{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"172.18.0.4","port":5432},"msg":"connect ECONNREFUSED 172.18.0.4:5432","time":"2024-04-25T16:00:59.333Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":40,"module":"Db","msg":"pgSubscriber: Reconnecting attempt 1","time":"2024-04-25T16:00:59.334Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":40,"module":"Db","msg":"pgSubscriber: Reconnecting attempt 2","time":"2024-04-25T16:00:59.855Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"HasuraBackgroundClient","msg":"Initializing","time":"2024-04-25T16:00:59.876Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"HasuraBackgroundClient","msg":"Initialized","time":"2024-04-25T16:01:16.751Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"MetadataFetchClient","msg":"Initializing","time":"2024-04-25T16:01:16.751Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"MetadataFetchClient","msg":"Initialized","time":"2024-04-25T16:01:17.190Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"ChainFollower","msg":"Initializing","time":"2024-04-25T16:01:17.191Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"ChainFollower","msg":"Initialized","time":"2024-04-25T16:01:17.196Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"Worker","msg":"Starting","time":"2024-04-25T16:01:17.196Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"Worker","msg":"Started","time":"2024-04-25T16:01:17.292Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"ChainFollower","msg":"Starting","time":"2024-04-25T16:01:17.296Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"ChainFollower","msg":"Started","time":"2024-04-25T16:01:17.375Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"ChainFollower","msg":"Rolling back to genesis","time":"2024-04-25T16:01:17.378Z","v":0}
{"name":"background","hostname":"7640fef91ae8","pid":1,"level":30,"module":"ChainFollower","msg":"Deleted 0 assets","time":"2024-04-25T16:01:17.381Z","v":0}
Inserted epoch 12 from updateEpochWhenSyncing with Cache. 
 epoch: Epoch {epochOutSum = 17487299670370, epochFees = DbLovelace 692317438, epochTxCount = 2569, epochBlkCount = 4226, epochNo = 12, epochStartTime = 2022-11-06 00:00:00 UTC, epochEndTime = 2022-11-06 23:59:45 UTC}
[db-sync-node:Info:72] [2024-04-25 16:02:35.72 UTC] Asynchronously wrote a ledger snapshot to /var/lib/cexplorer/1123185-6070bd7d3b-12.lstate in 0.020929209s.
[db-sync-node:Info:64] [2024-04-25 16:02:37.09 UTC] Inserted 129 EpochStake for EpochNo 14
[db-sync-node:Info:64] [2024-04-25 16:02:49.64 UTC] Insert Babbage Block: epoch 13, slot 1207360, block 55000, hash 607ff052f78a4c862139104aac114114ac0e385cf76a08a183a9e3ae08e75a41
[db-sync-node:Info:64] [2024-04-25 16:02:49.88 UTC] Removing old epoch boundary file /var/lib/cexplorer/604792-f71e518e4a-6.lstate
[db-sync-node:Info:64] [2024-04-25 16:02:49.88 UTC] Inserted 1 deposit refund rewards
[db-sync-node:Info:64] [2024-04-25 16:02:49.88 UTC] Removing 1 orphaned rewards
[db-sync-node:Info:64] [2024-04-25 16:02:49.88 UTC] Validate Epoch Rewards: total rewards that become spendable in epoch 14 are 59
[db-sync-node:Info:64] [2024-04-25 16:02:49.88 UTC] Persistant SQL Statement Cache size is 305
[db-sync-node:Info:64] [2024-04-25 16:02:49.88 UTC] 
Cache Statistics:
  Stake Addresses: cache size: 151, hit rate: 31%, hits: 6551, misses: 14481
  Pools: cache size: 53, hit rate: 84%, hits: 47513, misses: 8693
  Datums: cache capacity: 50000, cache size: 1527, hit rate: 49%, hits: 1477, misses: 1527
  Multi Assets: cache capacity: 250000, cache size: 1263, hit rate: 96%, hits: 77727, misses: 2526
  Previous Block: hit rate: 49%, hits: 55112, misses: 55113
[db-sync-node:Info:64] [2024-04-25 16:02:49.88 UTC] Starting epoch 14
[db-sync-node:Info:64] [2024-04-25 16:02:49.88 UTC] Insert Babbage Block: epoch 14, slot 1209602, block 55113, hash 43e71c0405e4f26c28763074def8843f87fd9cc2f4203227ab5bf0d976fecf3e
[db-sync-node:Info:64] [2024-04-25 16:02:49.89 UTC] 
Inserted epoch 13 from updateEpochWhenSyncing with Cache. 
 epoch: Epoch {epochOutSum = 35324831785160, epochFees = DbLovelace 1661774379, epochTxCount = 6396, epochBlkCount = 4173, epochNo = 13, epochStartTime = 2022-11-07 00:00:10 UTC, epochEndTime = 2022-11-07 23:59:58 UTC}
[db-sync-node:Info:72] [2024-04-25 16:02:49.90 UTC] Asynchronously wrote a ledger snapshot to /var/lib/cexplorer/1209598-e2d185967f-13.lstate in 0.028262778s.
[db-sync-node:Info:64] [2024-04-25 16:02:50.53 UTC] Inserted 146 EpochStake for EpochNo 15
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"172.18.0.3","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"query_execution_time":9.335416e-3,"request_id":"ccee2f69-17f4-4694-be3a-3ad5a11cbe9f","request_mode":"single","request_read_time":7.08e-7,"response_size":122,"uncompressed_response_size":122,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"ccee2f69-17f4-4694-be3a-3ad5a11cbe9f"},"level":"info","timestamp":"2024-04-25T16:03:18.962+0000","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"172.18.0.3","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"query_execution_time":1.3822125e-2,"request_id":"c6e0315d-3ad3-4ff8-9d11-e285a06ad065","request_mode":"single","request_read_time":5.41e-7,"response_size":66,"uncompressed_response_size":66,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"c6e0315d-3ad3-4ff8-9d11-e285a06ad065"},"level":"info","timestamp":"2024-04-25T16:03:18.962+0000","type":"http-log"}
Running
{"name":"cardano-graphql","hostname":"c2fcd8153573","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 94.42% | Asset: 0%","time":"2024-04-25T16:04:14.030Z","v":0}
{"name":"cardano-graphql","hostname":"c2fcd8153573","pid":1,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-04-25T16:04:18.988Z","v":0}
{"name":"cardano-graphql","hostname":"c2fcd8153573","pid":1,"level":30,"module":"CardanoNodeClient","tip":{"slot":1776638,"id":"ff1d1f9a69ee978dedcb8fbf2ed24718cb94ef403610077471528f3a2b1eb51f"},"msg":"[Object: null prototype] {\n  slot: 1776638,\n  id: 'ff1d1f9a69ee978dedcb8fbf2ed24718cb94ef403610077471528f3a2b1eb51f'\n}","time":"2024-04-25T16:04:18.997Z","v":0}
{"name":"cardano-graphql","hostname":"c2fcd8153573","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 94.45% | Asset: 0%","time":"2024-04-25T16:04:19.007Z","v":0}
[57125f53:cardano.node.ChainDB:Notice:32] [2024-04-25 16:04:55.93 UTC] Chain extended, new tip: 94dd3b5161c5cfbab56eaa5a527080ffbdcaa9f65c32cdf6f6ee9ff69115ebdd at slot 1939376
[57125f53:cardano.node.ChainDB:Notice:32] [2024-04-25 16:04:56.01 UTC] Chain extended, new tip: 1775b0a15bda382cf0421de65c51b46d2f35f4371b21bf02419ce83bc8b775fe at slot 1939921
[57125f53:cardano.node.ChainDB:Notice:32] [2024-04-25 16:04:56.70 UTC] Chain extended, new tip: e8e2c8b8d15da318bfaab87e5f83b5afb9090e0ebb376f1de1fd99e54c194a33 at slot 1944059
[57125f53:cardano.node.ChainDB:Info:32] [2024-04-25 16:04:57.18 UTC] Valid candidate ba195157a5471e1431649319649f46cd3b9b489807cf51ea97beb45dd16ed70c at slot 1946739
[57125f53:cardano.node.ChainDB:Notice:32] [2024-04-25 16:04:57.18 UTC] Chain extended, new tip: ba195157a5471e1431649319649f46cd3b9b489807cf51ea97beb45dd16ed70c at slot 1946739
[57125f53:cardano.node.ChainDB:Notice:32] [2024-04-25 16:04:57.40 UTC] Chain extended, new tip: 09b0c28db8c0bc1bb4500e4f84b0e40372474b36fb3782a83654d679667fad18 at slot 1948069
saravadeanil commented 4 months ago

Additionally, do I need to make any further changes to run the cardano-node 8.7.3 LINK component with the above ? @Kammerlo

Kammerlo commented 4 months ago

Hey, yes this looks perfect! I'm glad it worked out for you. The cardano node 8.7.3 is already running within the ogmios container. So no further changes needed.

saravadeanil commented 4 months ago

@Kammerlo Thanks a lot for you help : )

saravadeanil commented 4 months ago

@Kammerlo I am able to run all the cardano components.

However now after a while the cardano-graphql-background container crashes with below error:

{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"Db","msg":"Initializing...","time":"2024-04-26T16:04:46.000Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"HasuraBackgroundClient","msg":"Initializing","time":"2024-04-26T16:04:46.009Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"HasuraBackgroundClient","msg":"Initialized","time":"2024-04-26T16:04:52.161Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"MetadataFetchClient","msg":"Initializing","time":"2024-04-26T16:04:52.162Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"MetadataFetchClient","msg":"Initialized","time":"2024-04-26T16:04:52.957Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"ChainFollower","msg":"Initializing","time":"2024-04-26T16:04:52.957Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"ChainFollower","msg":"Initialized","time":"2024-04-26T16:04:52.993Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"Worker","msg":"Starting","time":"2024-04-26T16:04:52.993Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"Worker","msg":"Started","time":"2024-04-26T16:04:53.026Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"ChainFollower","msg":"Starting","time":"2024-04-26T16:04:53.032Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"ChainFollower","msg":"Started","time":"2024-04-26T16:04:53.095Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"ChainFollower","msg":"Rolling back to genesis","time":"2024-04-26T16:04:53.096Z","v":0}
{"name":"background","hostname":"cardano-temp-1-0","pid":7,"level":30,"module":"ChainFollower","msg":"Deleted 0 assets","time":"2024-04-26T16:04:53.109Z","v":0}
/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:85
                                    for (const entry of Object.entries(tx.mint.assets)) {
                                                               ^

TypeError: Cannot convert undefined or null to object
    at Function.entries (<anonymous>)
    at ChainFollower.<anonymous> (/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:85:64)
    at Generator.next (<anonymous>)
    at /app/packages/api-cardano-db-hasura/dist/ChainFollower.js:31:71
    at new Promise (<anonymous>)
    at __awaiter (/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:27:12)
    at Object.rollForward (/app/packages/api-cardano-db-hasura/dist/ChainFollower.js:80:62)
    at handler (/app/node_modules/@cardano-ogmios/client/dist/ChainSynchronization/nextBlock.js:23:46)
    at messageHandler (/app/node_modules/@cardano-ogmios/client/dist/ChainSynchronization/Client.js:17:43)
    at asyncWrapper (/app/node_modules/fastq/queue.js:213:12)

Node.js v18.19.1
2024-04-26 15:52:42.594 UTC [132] LOG:  could not receive data from client: Connection reset by peer
2024-04-26 16:01:22.499 UTC [181] ERROR:  relation "public.Epoch" does not exist at character 288
2024-04-26 16:01:22.499 UTC [181] STATEMENT:  SELECT  coalesce(json_agg("root" ORDER BY "root.pg.number" DESC NULLS FIRST), '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.base"."number" AS "number"       ) AS "_e"      ) ) AS "root", "_root.base"."number" AS "root.pg.number" FROM  (SELECT  *  FROM "public"."Epoch"  WHERE ('true')   ORDER BY "number" DESC NULLS FIRST LIMIT 1 ) AS "_root.base"    ORDER BY "root.pg.number" DESC NULLS FIRST  ) AS "_root"      
2024-04-26 16:01:22.593 UTC [181] ERROR:  relation "public.Cardano" does not exist at character 199
2024-04-26 16:01:22.593 UTC [181] STATEMENT:  SELECT  coalesce(json_agg("root" ), '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.tip"."tip" AS "tip"       ) AS "_e"      ) ) AS "root" FROM  (SELECT  *  FROM "public"."Cardano"  WHERE ('true')    LIMIT 1 ) AS "_root.base" LEFT OUTER JOIN LATERAL (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.tip.base"."number" AS "number"       ) AS "_e"      ) ) AS "tip" FROM  (SELECT  *  FROM "public"."Block"  WHERE (("_root.base"."tipBlockNo") = ("number"))    LIMIT 1 ) AS "_root.or.tip.base"      ) AS "_root.or.tip" ON ('true')      ) AS "_root"      
2024-04-26 16:01:22.692 UTC [181] ERROR:  relation "public.Cardano" does not exist at character 199
2024-04-26 16:01:22.692 UTC [181] STATEMENT:  SELECT  coalesce(json_agg("root" ), '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.tip"."tip" AS "tip"       ) AS "_e"      ) ) AS "root" FROM  (SELECT  *  FROM "public"."Cardano"  WHERE ('true')    LIMIT 1 ) AS "_root.base" LEFT OUTER JOIN LATERAL (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.tip.base"."number" AS "number"       ) AS "_e"      ) ) AS "tip" FROM  (SELECT  *  FROM "public"."Block"  WHERE (("_root.base"."tipBlockNo") = ("number"))    LIMIT 1 ) AS "_root.or.tip.base"      ) AS "_root.or.tip" ON ('true')      ) AS "_root"      
2024-04-26 16:04:47.509 UTC [202] ERROR:  relation "public.TokenMint" does not exist at character 154
2024-04-26 16:04:47.509 UTC [202] STATEMENT:  SELECT  json_build_object('aggregate', json_build_object('count', COUNT(*) ) ) AS "root" FROM  (SELECT  1  FROM  (SELECT DISTINCT ON ("assetId") *  FROM "public"."TokenMint"  WHERE ('true')     ) AS "_root.base"      ) AS "_root"      
2024-04-26 16:04:47.597 UTC [202] ERROR:  relation "public.Epoch" does not exist at character 288
2024-04-26 16:04:47.597 UTC [202] STATEMENT:  SELECT  coalesce(json_agg("root" ORDER BY "root.pg.number" DESC NULLS FIRST), '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.base"."number" AS "number"       ) AS "_e"      ) ) AS "root", "_root.base"."number" AS "root.pg.number" FROM  (SELECT  *  FROM "public"."Epoch"  WHERE ('true')   ORDER BY "number" DESC NULLS FIRST LIMIT 1 ) AS "_root.base"    ORDER BY "root.pg.number" DESC NULLS FIRST  ) AS "_root"      
2024-04-26 16:08:02.579 UTC [229] ERROR:  relation "public.TokenMint" does not exist at character 154
2024-04-26 16:08:02.579 UTC [229] STATEMENT:  SELECT  json_build_object('aggregate', json_build_object('count', COUNT(*) ) ) AS "root" FROM  (SELECT  1  FROM  (SELECT DISTINCT ON ("assetId") *  FROM "public"."TokenMint"  WHERE ('true')     ) AS "_root.base"      ) AS "_root"      
2024-04-26 16:08:02.601 UTC [229] ERROR:  relation "public.Epoch" does not exist at character 288
2024-04-26 16:08:02.601 UTC [229] STATEMENT:  SELECT  coalesce(json_agg("root" ORDER BY "root.pg.number" DESC NULLS FIRST), '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.base"."number" AS "number"       ) AS "_e"      ) ) AS "root", "_root.base"."number" AS "root.pg.number" FROM  (SELECT  *  FROM "public"."Epoch"  WHERE ('true')   ORDER BY "number" DESC NULLS FIRST LIMIT 1 ) AS "_root.base"    ORDER BY "root.pg.number" DESC NULLS FIRST  ) AS "_root"
curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "{ cardanoDbMeta { initialized syncPercentage }}"}' \
  http://localhost:3100/graphql | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

{
  "data": {
    "cardanoDbMeta": {
      "initialized": false,
      "syncPercentage": 100
    }
  }
}

curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "{ cardano { tip { number slotNo epoch { number } } } }"}' http://localhost:3100/graphql | jq .

{
  "data": {
    "cardano": {
      "tip": {
        "number": 61561,
        "slotNo": "1342742",
        "epoch": null
      }
    }
  }
}

cardano-cli query tip --testnet-magic 2 --socket-path /ipc/node.socket

{
    "block": 62868,
    "epoch": 15,
    "era": "Babbage",
    "hash": "873f1b2df70804ea72a6c8db0edfedcbfcd08b7d68d808f7d554939a80e3c308",
    "slot": 1369552,
    "slotInEpoch": 73552,
    "slotsToEpochEnd": 12848,
    "syncProgress": "2.89"
}
YuXiaoCoder commented 4 months ago

I'm getting a similar error running on the mainnet, do you have a similar table and view in your database? image

2024-04-28 02:27:33.304 UTC [10984] ERROR:  relation "public.TokenMint" does not exist at character 154
2024-04-28 02:27:33.304 UTC [10984] STATEMENT:  SELECT  json_build_object('aggregate', json_build_object('count', COUNT(*) ) ) AS "root" FROM  (SELECT  1  FROM  (SELECT DISTINCT ON ("assetId") *  FROM "public"."TokenMint"  WHERE ('true')     ) AS "_root.base"      ) AS "_root"

image

Kammerlo commented 4 months ago

@saravadeanil Hey, sorry for the delay with this issue. @YuXiaoCoder Opened the same issue so I'm currently working on both and I will fix this for you. I already fixed the error why the chainfollower crashes. I will implement everything regarding this in this branch: fix/syncing-issues-assets-missing So let me know if you encounter other things.

saravadeanil commented 4 months ago

I'm getting a similar error running on the mainnet, do you have a similar table and view in your database?

Hi @YuXiaoCoder, Even though I'm running on the preview network, yes, the issue situation is the same.

saravadeanil commented 4 months ago

@saravadeanil Hey, sorry for the delay with this issue. @YuXiaoCoder Opened the same issue so I'm currently working on both and I will fix this for you. I already fixed the error why the chainfollower crashes. I will implement everything regarding this in this branch: fix/syncing-issues-assets-missing So let me know if you encounter other things.

@Kammerlo No worries, thanks a lot : ))

saravadeanil commented 4 months ago

Hi @Kammerlo Based on the cardano-node-ogmios sync status, it appears to be catching up with the latest block height LINK. However, when running the GraphQL query, I am encountering the following errors:

Do you think it is related to above issue?

cardano-cli query tip --testnet-magic 2 --socket-path /ipc/node.socket
{
    "block": 2041926,
    "epoch": 553,
    "era": "Babbage",
    "hash": "d41d119fff94506865a70fc7ea33e415af6645c1559a506615dca93127914204",
    "slot": 47809531,
    "slotInEpoch": 30331,
    "slotsToEpochEnd": 56069,
    "syncProgress": "100.00"
}

curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "{ cardanoDbMeta { initialized syncPercentage }}"}' \
  http://localhost:3100/graphql | jq .

{
  "errors": [
    {
      "message": "field 'epochs' not found in type: 'query_root': {\"response\":{\"errors\":[{\"extensions\":{\"code\":\"validation-failed\",\"path\":\"$.selectionSet.epochs\"},\"message\":\"field 'epochs' not found in type: 'query_root'\"}],\"status\":200},\"request\":{\"query\":\"query {\\n          epochs (limit: 1, order_by: { number: desc }) {\\n              number\\n          }\\n          cardano {\\n              tip {\\n                  epoch {\\n                      number\\n                  }\\n                  slotNo\\n                  forgedAt\\n              }\\n          }}\"}}",
      "locations": [
        {
          "line": 1,
          "column": 3
        }
      ],
      "path": [
        "cardanoDbMeta"
      ],
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR",
        "exception": {
          "response": {
            "errors": [
              {
                "extensions": {
                  "code": "validation-failed",
                  "path": "$.selectionSet.epochs"
                },
                "message": "field 'epochs' not found in type: 'query_root'"
              }
            ],
            "status": 200
          },
          "request": {
            "query": "query {\n          epochs (limit: 1, order_by: { number: desc }) {\n              number\n          }\n          cardano {\n              tip {\n                  epoch {\n                      number\n                  }\n                  slotNo\n                  forgedAt\n              }\n          }}"
          },
          "stacktrace": [
            "Error: field 'epochs' not found in type: 'query_root': {\"response\":{\"errors\":[{\"extensions\":{\"code\":\"validation-failed\",\"path\":\"$.selectionSet.epochs\"},\"message\":\"field 'epochs' not found in type: 'query_root'\"}],\"status\":200},\"request\":{\"query\":\"query {\\n          epochs (limit: 1, order_by: { number: desc }) {\\n              number\\n          }\\n          cardano {\\n              tip {\\n                  epoch {\\n                      number\\n                  }\\n                  slotNo\\n                  forgedAt\\n              }\\n          }}\"}}",
            "    at GraphQLClient.<anonymous> (/app/node_modules/graphql-request/dist/index.js:170:35)",
            "    at step (/app/node_modules/graphql-request/dist/index.js:63:23)",
            "    at Object.next (/app/node_modules/graphql-request/dist/index.js:44:53)",
            "    at fulfilled (/app/node_modules/graphql-request/dist/index.js:35:58)",
            "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          ]
        }
      }
    }
  ],
  "data": null
}
curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "{ cardano { tip { number slotNo epoch { number } } } }"}' http://localhost:3100/graphql | jq .

{
  "errors": [
    {
      "message": "Cannot return null for non-nullable field Query.cardano.",
      "locations": [
        {
          "line": 1,
          "column": 3
        }
      ],
      "path": [
        "cardano"
      ],
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR",
        "exception": {
          "stacktrace": [
            "Error: Cannot return null for non-nullable field Query.cardano.",
            "    at completeValue (/app/node_modules/graphql/execution/execute.js:560:13)",
            "    at /app/node_modules/graphql/execution/execute.js:492:16",
            "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
            "    at async Promise.all (index 0)"
          ]
        }
      }
    }
  ],
  "data": null
}
Kammerlo commented 4 months ago

@saravadeanil Hi, I just tested it. (So it isn't currently at the tip, but still worth testing). I started it with NETWORK=preprod docker compose up -d --build and both your curls running fine:

So I can't reproduce your errors. How do you start it? Perhaps there's a difference?

saravadeanil commented 4 months ago

@Kammerlo I tried to deploy on preview network.

DOCKER_BUILDKIT=1 \
COMPOSE_DOCKER_CLI_BUILD=1 \
NETWORK=preview \
API_PORT=3102 \
HASURA_PORT=8092 \
OGMIOS_PORT=1339 \
POSTGRES_PORT=5434 \
METADATA_SERVER_URI="https://metadata.world.dev.cardano.org" \
docker compose -p preview up -d --build &&\
docker compose -p preview logs -f
saravadeanil commented 4 months ago

@Kammerlo It seems the endpoints are working now 🎉 Sorry for the noise.

cardano-cli query tip --testnet-magic 2 --socket-path /ipc/node.socket
{
    "block": 1643072,
    "epoch": 440,
    "era": "Babbage",
    "hash": "f6e41191157c0d4605a73d70f61218f9c14ec3e6410330a0a5d0289c4b28fefb",
    "slot": 38077178,
    "slotInEpoch": 61178,
    "slotsToEpochEnd": 25222,
    "syncProgress": "79.36"
}

~ curl   -X POST   -H "Content-Type: application/json"   -d '{"query": "{ cardanoDbMeta { initialized syncPercentage }}"}'   http://localhost:3100/graphql

{"data":{"cardanoDbMeta":{"initialized":true,"syncPercentage":100}}}

~ curl -X POST -H "Content-Type: application/json" -d '{"query": "{ cardano { tip { number slotNo epoch { number } } } }"}' http://localhost:3100/graphql

{"data":{"cardano":{"tip":{"number":2046519,"slotNo":"47922870","epoch":{"number":554}}}}}
Kammerlo commented 4 months ago

Awesome :) And don't be sorry for the noise, I'm glad we were able to fix everything and you can continue to use it!

So we can close this issue?

saravadeanil commented 4 months ago

@Kammerlo Thanks. I think the node is syncing, although it may take some time for the node to fully sync to the latest block height. For now, I think we are good. Thanks once again.

Additionally, just to double-check, will you be creating a new patch release for this ISSUE?

Kammerlo commented 4 months ago

Yes I will merge it soon

Kammerlo commented 4 months ago

Closing this issue since the bugfix is now in PreRelease 8.0.2 If you have any other issues, let me know.