Open mkocur-ledger opened 1 month ago
Kartiiyer12
commented
1 month ago Hi @mkocur-ledger I think the issue for the Epoch and withdrawal table missing could be issue with the hasura migration and not the db-sync-snapshot. Can you try to see if there are errors on the graphql-background & hasura while applying the migrations?
For the token metadata registry server, are you running the default container as it is from the repo? Table missing seems improbable.
mkocur-ledger
commented
1 month ago Hi @Kartiiyer12, thanks for your interest in my bugreport. About metadata, I have edit my report and removed it. It was fixed here: https://github.com/cardano-foundation/cf-token-metadata-registry/issues/19
About the migration, I didn't create any of my own. These are related log messages:
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_D
ATABASE_URL"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"migrations server port env var is not set, defaulting
to 9691"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"server timeout is not set, defaulting to 30 seconds"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9691"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 30 for 9691 to be ready"}}
{"detail":{"info":"Already at the latest catalog version (48); nothing to do.","kind":"catalog_migrate"},"level":"info","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"detail":{"info":"Schema sync enabled. Polling at Refined (Milliseconds {milliseconds = 1s})","kind":"schema-sync"},"level":"info","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"detail":{"info":{"instance_id":"4d2b28e4-c896-465b-88f0-3120b600e68c","message":"listener thread started","thread_id":"ThreadId 72"},"kind":"schema-sync"},"level":"info","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_MIGRATIONS_DIR is not set, defaulting to /hasura-migrations"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}These are migration folder that should be applied - https://github.com/cardano-foundation/cardano-graphql/tree/16ed6d71dbe95ff9e7d959ea51a8476da668684c/packages/api-cardano-db-hasura/hasura/project/migrations/default/1589369664961_init. This happens from the background-graphql side. Are you running everything as it is from the repo or are there any customizations?
mkocur-ledger
commented
1 month ago I have made a fresh installation and after applying this: https://github.com/cardano-foundation/cf-token-metadata-registry/blob/47f6cf08515f45c3dc11968a7e15a791875cf59c/api/src/main/resources/db/migration/postgresql/V0__metadata_server_db_init.sql
I see no errors. Created an issue that it is not applied automatically: https://github.com/cardano-foundation/cf-token-metadata-registry/issues/19
About your question, there are customizations, yes, but I can see messages in background pod that it is actually inserting assets and the assets were synced in graphql to 100%, so I assume it works.
Kammerlo
commented
1 month ago I'll check the issue in the token-registry next week and fix this. Thanks for finding it! And I'm glad it worked for you. I close this issue since it's probably a topic for the Token Registry. If there is anything else, let us know.
mkocur-ledger
commented
1 month ago Actually @Kammerlo please reopen this. I have disabled INFO messages and I found those errors again (I have applied the sql file you mentioned on my database but it didn't help):
"relation \"public.Epoch\" does not exist"
relation \"public.Withdrawal\" does not exist"
{"error":{"code":"validation-failed","error":"no mutations exist","path":"$"}
{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"}Full log:
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":
"warn","timestamp":"2024-10-01T08:14:36.932+0000","type":"startup"} {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not
exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.base\".\"number\" AS \"number\" ) AS \"_e\" ) ) A
S \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":n
ull},"query_execution_time":59.342454474,"request_id":"858e9742-dc65-4e28-8906-a6259378869e","request_mode":"error","request_read_time":2.06e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"858e9742-dc65-4e28-8906-a625
9378869e"},"level":"error","span_id":"1856f494e02ce310","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"f070f1320c096d4104734fc7e6db5fa0","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Withdrawal\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT json_build_object('aggregate', json_build_object('sum', json_build_obje
ct('amount', sum(\"_root\".\"root.pg.amount\" ) ) ) ) AS \"root\" FROM (SELECT \"_root.base\".\"amount\" AS \"root.pg.amount\" FROM (SELECT * FROM \"public\".\"Withdrawal\" WHERE ('true') ) AS \"_root.base\" ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_e
xecution_time":65.628226835,"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766","request_mode":"error","request_read_time":1.54e-6,"response_size":669,"uncompressed_response_size":669,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766"},"l
evel":"error","span_id":"93a4c54f8be46366","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"0a44ad3329a1255d3af62717187eee0a","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not
exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.base\".\"number\" AS \"number\" ) AS \"_e\" ) ) A
S \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":n
ull},"query_execution_time":1.759961e-3,"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fedb441c0","request_mode":"error","request_read_time":2.38e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fe
db441c0"},"level":"error","span_id":"415b0c75b73d3f9b","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"7221d21491e389cadda32b981deeddc9","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[
]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.base\".\"number\" AS \"number\" ) AS \"_e\" ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DE
SC NULLS FIRST LIMIT 1 ) AS \"_root.base\" ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_execution_time":1.466746e-3,"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67","request_mode":"error","request_read_time":8.1e-7,"r
esponse_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67"},"level":"error","span_id":"f0fc2276bc19e1ad","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"f13c31a79f12fd3a95fea03cca50d56f","
type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[
]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.base\".\"number\" AS \"number\" ) AS \"_e\" ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DE
SC NULLS FIRST LIMIT 1 ) AS \"_root.base\" ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_execution_time":1.23195e-3,"request_id":"dc69b318-34f9-4282-becb-0e487414eac3","request_mode":"error","request_read_time":1.01e-6,"r
esponse_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"dc69b318-34f9-4282-becb-0e487414eac3"},"level":"error","span_id":"83934b4c0740d43f","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"d4b99c418de9e6fcca17a14586011e69","
type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[
]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.or.adaPots\".\"adaPots\" AS \"adaPots\" ) AS \"_e\" ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"num
ber\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" LEFT OUTER JOIN LATERAL (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.or.adaPots.base\".\"reserves\" AS \"reserves\" ) AS \"_e\" ) ) AS \"adaPots\" FROM (SELECT * FROM \"public\".\"ada_pots\" WHERE ((\"_root
.base\".\"number\") = (\"epoch_no\")) LIMIT 1 ) AS \"_root.or.adaPots.base\" ) AS \"_root.or.adaPots\" ON ('true') ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_execution_time":1.229809e-3,"request_id":"5f82fa48-a
433-4b54-af2a-ef488244b533","request_mode":"error","request_read_time":1.07e-6,"response_size":1194,"uncompressed_response_size":1194,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"5f82fa48-a433-4b54-af2a-ef488244b533"},"level":"error","span_id":"ca4807fd288b05b5","timest
amp":"2024-10-01T12:55:28.255+0000","trace_id":"33c6aa0f5dda79245bc95b61916ef6e5","type":"http-log"} {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not
exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.base\".\"number\" AS \"number\" ) AS \"_e\" ) ) A
S \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":n
ull},"query_execution_time":1.657673e-3,"request_id":"2fc31b05-0f3c-4dd6-9192-60204fbb4c8d","request_mode":"error","request_read_time":1.72e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"2fc31b05-0f3c-4dd6-9192-60204
fbb4c8d"},"level":"error","span_id":"95066c5648e8a6ec","timestamp":"2024-10-01T12:55:32.259+0000","trace_id":"671b908ec3a31d38c24227731b82c72e","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.122","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"no mutations exist","path":"$"},"query":{"type":null},"query_execution_time":1.67017e-
4,"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a","request_mode":"error","request_read_time":1.66637e-4,"response_size":68,"uncompressed_response_size":68,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a"},"level":"error","span_id":"
e2a38977c75f3546","timestamp":"2024-10-01T12:55:33.259+0000","trace_id":"ba50a6997116f6e3be10e397d77dc9d6","type":"http-log"} {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.50166e-4,"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55","request_mode":"error","request_read_time":
2.36e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55"},"level":"error","span_id":"e8d20c1563873073","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"2fe4e636804b493c12da50d98
4004e3c","type":"http-log"} {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.43936e-4,"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95","request_mode":"error","request_read_time":
2.06e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95"},"level":"error","span_id":"5d374fc3c09d8b7a","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"0416d3597fb9ebc592dbaffe7
483147d","type":"http-log"}Could you provide logs from hasura service? Because this service should apply automatically the sql files and create these tables.
Kammerlo
commented
1 month ago @mkocur-ledger I further investigated what could be the problem. I found an error within the sql file, which hindert hasura to finish the migration. You can find the PR here: https://github.com/cardano-foundation/cardano-graphql/pull/898 I'll finish testing today, merge and build a new fix release.
Could you check if this solves the problems for your issue as well?
mkocur-ledger
commented
1 month ago @Kammerlo I have seen this, I was actually supposed to create a PR or bugreport, but didn't find time yet.
Actually, this is not going to help, I have run those SQL queries already against the database (fixing the missing comma).
// full logs as requested:
{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_DATABASE_URL"}}
{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"migrations server port env var is not set, defaulting to 9691"}}
{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9691"}}
{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 60 for 9691 to be ready"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-01T08:14:35.949+0000","type":"startup"}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"port 9691 is ready"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_MIGRATIONS_DIR is not set, defaulting to /hasura-migrations"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-01T08:14:36.932+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-01T08:14:35.949+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-01T08:14:36.932+0000","type":"startup"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.base\".\"number\" AS \"number\" ) AS \"_e\" ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_execution_time":59.342454474,"request_id":"858e9742-dc65-4e28-8906-a6259378869e","request_mode":"error","request_read_time":2.06e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"858e9742-dc65-4e28-8906-a6259378869e"},"level":"error","span_id":"1856f494e02ce310","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"f070f1320c096d4104734fc7e6db5fa0","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Withdrawal\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT json_build_object('aggregate', json_build_object('sum', json_build_object('amount', sum(\"_root\".\"root.pg.amount\" ) ) ) ) AS \"root\" FROM (SELECT \"_root.base\".\"amount\" AS \"root.pg.amount\" FROM (SELECT * FROM \"public\".\"Withdrawal\" WHERE ('true') ) AS \"_root.base\" ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_execution_time":65.628226835,"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766","request_mode":"error","request_read_time":1.54e-6,"response_size":669,"uncompressed_response_size":669,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766"},"level":"error","span_id":"93a4c54f8be46366","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"0a44ad3329a1255d3af62717187eee0a","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.base\".\"number\" AS \"number\" ) AS \"_e\" ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_execution_time":1.759961e-3,"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fedb441c0","request_mode":"error","request_read_time":2.38e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fedb441c0"},"level":"error","span_id":"415b0c75b73d3f9b","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"7221d21491e389cadda32b981deeddc9","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.base\".\"number\" AS \"number\" ) AS \"_e\" ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_execution_time":1.466746e-3,"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67","request_mode":"error","request_read_time":8.1e-7,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67"},"level":"error","span_id":"f0fc2276bc19e1ad","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"f13c31a79f12fd3a95fea03cca50d56f","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.base\".\"number\" AS \"number\" ) AS \"_e\" ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_execution_time":1.23195e-3,"request_id":"dc69b318-34f9-4282-becb-0e487414eac3","request_mode":"error","request_read_time":1.01e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"dc69b318-34f9-4282-becb-0e487414eac3"},"level":"error","span_id":"83934b4c0740d43f","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"d4b99c418de9e6fcca17a14586011e69","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.or.adaPots\".\"adaPots\" AS \"adaPots\" ) AS \"_e\" ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" LEFT OUTER JOIN LATERAL (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.or.adaPots.base\".\"reserves\" AS \"reserves\" ) AS \"_e\" ) ) AS \"adaPots\" FROM (SELECT * FROM \"public\".\"ada_pots\" WHERE ((\"_root.base\".\"number\") = (\"epoch_no\")) LIMIT 1 ) AS \"_root.or.adaPots.base\" ) AS \"_root.or.adaPots\" ON ('true') ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_execution_time":1.229809e-3,"request_id":"5f82fa48-a433-4b54-af2a-ef488244b533","request_mode":"error","request_read_time":1.07e-6,"response_size":1194,"uncompressed_response_size":1194,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"5f82fa48-a433-4b54-af2a-ef488244b533"},"level":"error","span_id":"ca4807fd288b05b5","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"33c6aa0f5dda79245bc95b61916ef6e5","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.base\".\"number\" AS \"number\" ) AS \"_e\" ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ('true') ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" ORDER BY \"root.pg.number\" DESC NULLS FIRST ) AS \"_root\" "},"path":"$"},"query":{"type":null},"query_execution_time":1.657673e-3,"request_id":"2fc31b05-0f3c-4dd6-9192-60204fbb4c8d","request_mode":"error","request_read_time":1.72e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"2fc31b05-0f3c-4dd6-9192-60204fbb4c8d"},"level":"error","span_id":"95066c5648e8a6ec","timestamp":"2024-10-01T12:55:32.259+0000","trace_id":"671b908ec3a31d38c24227731b82c72e","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.122","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"no mutations exist","path":"$"},"query":{"type":null},"query_execution_time":1.67017e-4,"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a","request_mode":"error","request_read_time":1.66637e-4,"response_size":68,"uncompressed_response_size":68,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a"},"level":"error","span_id":"e2a38977c75f3546","timestamp":"2024-10-01T12:55:33.259+0000","trace_id":"ba50a6997116f6e3be10e397d77dc9d6","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.50166e-4,"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55","request_mode":"error","request_read_time":2.36e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55"},"level":"error","span_id":"e8d20c1563873073","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"2fe4e636804b493c12da50d984004e3c","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.43936e-4,"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95","request_mode":"error","request_read_time":2.06e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95"},"level":"error","span_id":"5d374fc3c09d8b7a","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"0416d3597fb9ebc592dbaffe7483147d","type":"http-log"}//edit forgot I have warn level. Here's debug from start:
{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_DATABASE_URL"}}
{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"migrations server port env var is not set, defaulting to 9691"}}
{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9691"}}
{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 60 for 9691 to be ready"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"admin_secret_set":false,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"console_sentry_dsn":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":true,"enabled_apis":["metadata"],"enabled_log_types":["startup","http-log","jwk-refresh-log","webhook-log","websocket-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"debug","port":9691,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":false,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":null,"use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"Refined (Seconds {seconds = 3s})","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Refined (Seconds {seconds = 5s})}"},"kind":"server_configuration"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"database_url":"postgres://cardano-mainnet:...@10.99.2.57:5432/cardano-mainnet","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"port 9691 is ready"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_MIGRATIONS_DIR is not set, defaulting to /hasura-migrations"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"admin_secret_set":false,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"console_sentry_dsn":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":true,"enabled_apis":["metrics","metadata","config","graphql"],"enabled_log_types":["startup","http-log","jwk-refresh-log","webhook-log","websocket-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"debug","port":9090,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":false,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":null,"use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"Refined (Seconds {seconds = 3s})","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Refined (Seconds {seconds = 5s})}"},"kind":"server_configuration"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"database_url":"postgres://cardano-mainnet:...@10.99.2.57:5432/cardano-mainnet","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":"Thread runCronEventsGenerator (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread processScheduledTriggers (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"Unlocking all locked scheduled events on `hdb_scheduled_events` and `hdb_cron_events` tables","kind":"scheduled_triggers"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread checkForUpdates (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread websocketConnectionReaper (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"Already at the latest catalog version (48); nothing to do.","kind":"catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Schema sync enabled. Polling at Refined (Milliseconds {milliseconds = 1s})","kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"instance_id":"b580b3d4-6fb3-4180-bc52-22dfc2baba95","message":"listener thread started","thread_id":"ThreadId 72"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"message":"source \"default\" has not been initialized yet.","source":"default"},"kind":"source_catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"instance_id":"b580b3d4-6fb3-4180-bc52-22dfc2baba95","message":"processor thread started","thread_id":"ThreadId 95"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Starting in eventing enabled mode","kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Starting workers","kind":"event_triggers"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Help us improve Hasura! The graphql-engine server collects anonymized usage stats which allows us to keep improving Hasura at warp speed. To read more or opt-out, visit https://hasura.io/docs/latest/graphql/core/guides/telemetry.html","kind":"telemetry"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"message":"Starting API server","time_taken":0.394567301},"kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread SchemeUpdate.listener (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread SchemeUpdate.processor (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread processEventQueue (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread sourcePingPoller (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread runTelemetry (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"metrics_info: {\"server_metrics\":{\"resource_cpu\":4096,\"resource_memory\":10737418240,\"resource_checker_error_code\":null}}","level":"debug","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Shutting websockets server down","level":"debug","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal updateJWK thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal runTelemetry thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal websocketConnectionReaper thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal sourcePingPoller thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal checkForUpdates thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"All in-flight events have finished processing","kind":"scheduled_events"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread ourIdleGC (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread asyncActionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread updateJWK (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"Gracefully shutting down server","kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread asyncActionSubscriptionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread runCronEventsGenerator (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread processScheduledTriggers (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread checkForUpdates (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread sourcePingPoller (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":{"info":"Unlocking all locked scheduled events on `hdb_scheduled_events` and `hdb_cron_events` tables","kind":"scheduled_triggers"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Already at the latest catalog version (48); nothing to do.","kind":"catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Schema sync enabled. Polling at Refined (Milliseconds {milliseconds = 1s})","kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"instance_id":"e9c63890-18ac-4a6f-a0d1-623c32926af0","message":"listener thread started","thread_id":"ThreadId 72"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"message":"source \"default\" has not been initialized yet.","source":"default"},"kind":"source_catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"instance_id":"e9c63890-18ac-4a6f-a0d1-623c32926af0","message":"processor thread started","thread_id":"ThreadId 94"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Starting in eventing enabled mode","kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Starting workers","kind":"event_triggers"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Help us improve Hasura! The graphql-engine server collects anonymized usage stats which allows us to keep improving Hasura at warp speed. To read more or opt-out, visit https://hasura.io/docs/latest/graphql/core/guides/telemetry.html","kind":"telemetry"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"message":"Starting API server","time_taken":0.443591001},"kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":"Thread SchemeUpdate.listener (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread SchemeUpdate.processor (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread asyncActionSubscriptionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread websocketConnectionReaper (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread runTelemetry (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"metrics_info: {\"server_metrics\":{\"resource_cpu\":4096,\"resource_memory\":10737418240,\"resource_checker_error_code\":null}}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread ourIdleGC (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread processEventQueue (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread updateJWK (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread asyncActionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"http_success: {\"error\":\"invalid_payload\"}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":{"http_error":{"http_exception":null,"response":"{\"error\":\"invalid_payload\"}","status_code":400,"url":"https://telemetry.hasura.io/v1/http"},"message":"failed to post telemetry","type":"http_error"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"telemetry-log"}
{"detail":"metrics_info: {\"topic\":\"server_metrics_v2\",\"data\":{\"metadata_db_uid\":\"156d95bd-4660-480c-9b89-a7a37d9dd9ff\",\"instance_uid\":\"e9c63890-18ac-4a6f-a0d1-623c32926af0\",\"hasura_version\":\"v2.43.0\",\"ci\":null,\"source_metadata\":{\"db_uid\":\"156d95bd-4660-480c-9b89-a7a37d9dd9ff\",\"backend_type\":\"postgres\",\"db_kind\":\"postgres\",\"db_version\":\"150007\"},\"metrics\":{\"tables\":6,\"views\":27,\"enum_tables\":0,\"relationships\":{\"manual\":82,\"auto\":0},\"permissions\":{\"select\":32,\"insert\":1,\"update\":1,\"delete\":1,\"roles\":1},\"event_triggers\":0,\"functions\":0,\"remote_schemas\":0,\"service_timings\":{\"collectionTag\":1727872637,\"serviceTimingMetrics\":[]},\"actions\":{\"synchronous\":0,\"asynchronous\":0,\"query_actions\":0,\"type_relationships\":0,\"custom_types\":0},\"native_queries\":{\"with_parameters\":0,\"without_parameters\":0},\"stored_procedures\":{\"with_parameters\":0,\"without_parameters\":0},\"logical_models\":{\"count\":0}},\"experimental_features\":[]}}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"http_success: {\"message\":\"ok\"}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Stopping immortal processScheduledTriggers thread","level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"unstructured"}
{"detail":{"num_cron_events_fetched":0,"num_fetches":1,"num_one_off_scheduled_events_fetched":0},"level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"scheduled-trigger-process"}
{"detail":"Stopping immortal runCronEventsGenerator thread","level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"unstructured"}
{"detail":"Stopping immortal asyncActionSubscriptionsProcessor thread","level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"unstructured"}
{"detail":{"info":"All in-flight events have finished processing","kind":"async_actions"},"level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"startup"}
{"detail":"Stopping immortal asyncActionsProcessor thread","level":"info","timestamp":"2024-10-02T12:37:26.276+0000","type":"unstructured"}
{"detail":{"info":"All in-flight events have finished processing","kind":"event_triggers"},"level":"info","timestamp":"2024-10-02T12:37:26.276+0000","type":"startup"}
{"detail":"Stopping immortal processEventQueue thread","level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"unstructured"}
{"detail":"Stopping immortal SchemeUpdate.processor thread","level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"unstructured"}
{"detail":"Stopping immortal SchemeUpdate.listener thread","level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"unstructured"}
{"detail":{"num_events_fetched":{},"num_fetches":13},"level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"event-trigger-process"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"51.91.233.11","method":"GET","status":200,"url":"/healthz"},"operation":{"query":{"type":null},"request_id":"14b2edc0-899c-40b8-b84c-dd350104184b","request_mode":"non-graphql","response_size":2,"uncompressed_response_size":2},"request_id":"14b2edc0-899c-40b8-b84c-dd350104184b"},"level":"info","timestamp":"2024-10-02T12:37:38.258+0000","type":"http-log"}What about your background service? Is this one running properly?
mkocur-ledger
commented
1 month ago {"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"Db","msg":"Initializing...","time":"2024-10-02T13:03:43.963Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraBackgroundClient","msg":"Initializing","time":"2024-10-02T13:03:43.997Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraBackgroundClient","msg":"Initialized","time":"2024-10-02T13:03:51.805Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"MetadataFetchClient","msg":"Initializing","time":"2024-10-02T13:03:51.805Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"MetadataFetchClient","msg":"Metadata Server is up and running. Checking Sync Status.","time":"2024-10-02T13:03:51.813Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"MetadataFetchClient","msg":"Initialized","time":"2024-10-02T13:03:51.817Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"ChainFollower","msg":"Initializing","time":"2024-10-02T13:03:51.817Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"ChainFollower","msg":"Initialized","time":"2024-10-02T13:03:51.873Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"Worker","msg":"Starting","time":"2024-10-02T13:03:51.873Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"Worker","msg":"Started","time":"2024-10-02T13:03:52.018Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"id":"a650a3f398ba4a9427ec8c293e9f7156d81fd2f7ca849014d8d2c1156c359b3a","slot":23068800,"msg":"","time":"2024-10-02T13:03:52.018Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"8cfe22b2dd7708ff8ed5e65e5c8d8ce0a65365cc805b3ffa51946550444153","msg":"Adding metadata to asset","time":"2024-10-02T13:03:58.906Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"02b261951d13fddebeca13abe1ea9b096b6e4131364de1b24f32a23d41504f","msg":"Adding metadata to asset","time":"2024-10-02T13:03:58.921Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"e4214b7cce62ac6fbba385d164df48e157eae5863521b4b67ca71d867b4ccb2dd9de12c339a4aa099fe524342ffdc835be131122deacba92d9139f5b","msg":"Adding metadata to asset","time":"2024-10-02T13:03:58.938Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"605f3d2b1bdb339e0712c9b33be8df5da84c8fb8de969dacca5939b24d6f6d73","msg":"Adding metadata to asset","time":"2024-10-02T13:03:58.949Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"e4214b7cce62ac6fbba385d164df48e157eae5863521b4b67ca71d86df4422ce3d80fa783c1001758390c58e5f6a33c63d9622658a80e29565dcd1be","msg":"Adding metadata to asset","time":"2024-10-02T13:03:59.029Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"ed5517ccf67c60004355cee3c546c77226cd89a04b3aaeae6a65589e4d6f6d","msg":"Adding metadata to asset","time":"2024-10-02T13:03:59.044Z","v":0}
[...]@mkocur-ledger I doubled checked and tested the whole setup multiple times and couldn't reproduce the issue. Within your logs I see that the Hasurabackgroundclient is initialized. The schemas are created on this line https://github.com/cardano-foundation/cardano-graphql/blob/16ed6d71dbe95ff9e7d959ea51a8476da668684c/packages/api-cardano-db-hasura/src/HasuraBackgroundClient.ts#L59 which should be executed due to the log that it's initialized
Another Idea I want to check is if db-sync is running properly. Could you check the logs if there is something noticeable?
mkocur-ledger
commented
1 month ago Except what I found in the log at the very top of this ticket, not really. But that was fixed by adding metadata table manually.
Maybe you just miss those messages? I can see them when:
(BTW I start hasura, background, ogmios and graphql when db-sync reaches current epoch)
Kartiiyer12
commented
4 weeks ago Hi @mkocur-ledger Can you check in the database directly, the Epoch and the Withdrawal table exists? Also, what happens when you try to query the data via GraphQL for Epoch related data?
Summary
Hello!
I have restored a snapshot with db-sync: (https://update-cardano-mainnet.iohk.io/cardano-db-sync/13.5/db-sync-snapshot-schema-13.5-block-10823207-x86_64.tgz)
hasura/graphql-engine:v2.43.0.cli-migrations-v2 is reporting public.Withdrawal and public.Epoch relations missing:
The sync status:
I have restarted graphql, tmr, hasura, background components after the sync was completed to face the same issue.
db-sync: "13.5.0.2" node: "9.1.1" ogmios-server: "6.6.2" graphql-engine: "v2.43.0.cli-migrations-v2" graphql-server: "8.2.2" graphql-background: "8.2.2-mainnet" cf-token-metadata-registry-api: "1.0.2" postgres: 15.6
Steps to reproduce the bug
No response
Actual Result
Some db data missing after snapshot restore.
Expected Result
No errors
Environment
Kubernetes { os: { arch: 'x64', cpus: { count: 48, model: 'Intel(R) Xeon(R) Gold 6312U CPU @ 2.40GHz' }, release: '5.15.0-117-generic', totalMem: 269877899264, type: 'Linux' }, version: '8.0.0' }
Platform
Platform version
Ubuntu 22.04.4 LTS
Runtime
Runtime version
No response