search

cardano-foundation / cardano-graphql

GraphQL API for Cardano
Apache License 2.0
259 stars 103 forks source link

graphql database internal server error #878

Closed niso1985 closed 3 weeks ago

niso1985 commented 1 month ago

Summary

I have been using cardano-graphql 8.0.0. I updated to cardano-graphql 8.1.0 for Chang Hardfork.

Then only graphql query became frequently unstable.

Do you know what is causing this?

Steps to reproduce the bug

Changes are made to docker-compose.yml. See the attached docker-compose.yml.txt. docker-compose.yml.txt

cardano-db-sync was restored using

Environment=RESTORE_SNAPHOT=https://update-cardano-mainnet.iohk.io/cardano-db-sync/13.2/db-sync-snapshot-schema-13.2-block-10548081-x86_64.tgz

Actual Result

It takes more than 5 minutes to get this response

curl   -X POST   -H "Content-Type: application/json"   -d '{"query": "{ cardanoDbMeta { initialized syncPercentage }}"}'   http://localhost:3100/graphql

{
  "errors": [
    {
      "message": "database query error: {
        \"response\": {
          \"errors\": [
            {
              \"extensions\": {
                \"code\": \"unexpected\",
                \"path\": \"$\" 
              },
              \"message\": \"database query error\"
            }
          ],
          \"status\": 200
        },
        \"request\": {
          \"query\": \"query {
            epochs (limit: 1, order_by: { number: desc }) {
              number
            }
            cardano {
              tip {
                epoch {
                  number
                }
                slotNo
                forgedAt
              }
            }
          }\"
        }
      }",
      "locations": [
        {
          "line": 1,
          "column": 3
        }
      ],
      "path": [
        "cardanoDbMeta"
      ],
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR",
        "exception": {
          "response": {
            "errors": [
              {
                "extensions": {
                  "code": "unexpected",
                  "path": "$"
                },
                "message": "database query error"
              }
            ],
            "status": 200
          },
          "request": {
            "query": "query {
              epochs (limit: 1, order_by: { number: desc }) {
                number
              }
              cardano {
                tip {
                  epoch {
                    number
                  }
                  slotNo
                  forgedAt
                }
              }
            }\n          "
          },
          "stacktrace": [
            "Error: database query error: {
              \"response\": {
                \"errors\": [
                  {
                    \"extensions\": {
                      \"code\": \"unexpected\",
                      \"path\": \"$\" 
                    },
                    \"message\": \"database query error\"
                  }
                ],
                \"status\": 200
              },
              \"request\": {
                \"query\": \"query {
                  epochs (limit: 1, order_by: { number: desc }) {
                    number
                  }
                  cardano {
                    tip {
                      epoch {
                        number
                      }
                      slotNo
                      forgedAt
                    }
                  }
                }\"
              }
            }",
            "    at GraphQLClient.<anonymous> (/app/node_modules/graphql-request/dist/index.js:170:35)",
            "    at step (/app/node_modules/graphql-request/dist/index.js:63:23)",
            "    at Object.next (/app/node_modules/graphql-request/dist/index.js:44:53)",
            "    at fulfilled (/app/node_modules/graphql-request/dist/index.js:35:58)",
            "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          ]
        }
      }
    }
  ],
  "data": null
}

Expected Result

Sometimes this request is success.

curl   -X POST   -H "Content-Type: application/json"   -d '{"query": "{ cardanoDbMeta { initialized syncPercentage }}"}'   http://localhost:3100/graphql
{"data":{"cardanoDbMeta":{"initialized":true,"syncPercentage":100}}}

Environment

Amazon Linux 2/m6i.4xlarge EBS / : 300GB /data: 2.25TB

Platform

Platform version

Amazon Linux 2

Runtime

Runtime version

docker 20.10.7

Kammerlo commented 1 month ago

The current version isn't chang hardfork ready, since I need to wait for the db-sync upgrade and ogmios. But nevertheless this should work. Are there any log messages, which could help me getting closer to the issue? For example from the database or the graphql server.

niso1985 commented 1 month ago

Cardano-graphql-postgres is periodically outputting errors, which is also the case with preprod. preprod is working fine.

2024-07-13 05:03:19.509 UTC [18485] ERROR:  relation "public.Cardano" does not exist at character 199
2024-07-13 05:03:19.509 UTC [18485] STATEMENT:  SELECT  coalesce(json_agg("root" ), '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.tip"."tip" AS "tip"       ) AS "_e"      ) ) AS "root" FROM  (SELECT  *  FROM "public"."Cardano"  WHERE ('true')    LIMIT 1 ) AS "_root.base" LEFT OUTER JOINLATERAL (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.tip.base"."number" AS "number", ("_root.or.tip.base"."slotNo")::text AS "slotNo", "_root.or.tip.or.epoch"."epoch" AS "epoch"       ) AS "_e"      ) ) AS "tip" FROM  (SELECT  *  FROM "public"."Block"  WHERE (("_root.base"."tipBlockNo") = ("number"))    LIMIT 1 ) AS "_root.or.tip.base" LEFT OUTER JOIN LATERAL (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.tip.or.epoch.base"."number" AS "number"       ) AS "_e"      ) ) AS "epoch" FROM  (SELECT  *  FROM "public"."Epoch"  WHERE (("_root.or.tip.base"."epochNo") = ("number"))    LIMIT 1 ) AS "_root.or.tip.or.epoch.base"      ) AS "_root.or.tip.or.epoch" ON ('true')      ) AS "_root.or.tip" ON ('true')      ) AS "_root"
2024-07-13 05:03:19.509 UTC [18448] ERROR:  relation "public.TokenMint" does not exist at character 154
2024-07-13 05:03:19.509 UTC [18448] STATEMENT:  SELECT  json_build_object('aggregate', json_build_object('count', COUNT(*) ) ) AS "root" FROM  (SELECT  1  FROM  (SELECT DISTINCT ON ("assetId") *  FROM "public"."TokenMint"  WHERE ('true')     ) AS "_root.base"      ) AS "_root"
2024-07-13 05:03:19.509 UTC [18465] ERROR:  relation "public.Cardano" does not exist at character 199
2024-07-13 05:03:19.509 UTC [18465] STATEMENT:  SELECT  coalesce(json_agg("root" ), '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.tip"."tip" AS "tip"       ) AS "_e"      ) ) AS "root" FROM  (SELECT  *  FROM "public"."Cardano"  WHERE ('true')    LIMIT 1 ) AS "_root.base" LEFT OUTER JOINLATERAL (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.tip.base"."number" AS "number", ("_root.or.tip.base"."slotNo")::text AS "slotNo", "_root.or.tip.or.epoch"."epoch" AS "epoch"       ) AS "_e"      ) ) AS "tip" FROM  (SELECT  *  FROM "public"."Block"  WHERE (("_root.base"."tipBlockNo") = ("number"))    LIMIT 1 ) AS "_root.or.tip.base" LEFT OUTER JOIN LATERAL (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.tip.or.epoch.base"."number" AS "number"       ) AS "_e"      ) ) AS "epoch" FROM  (SELECT  *  FROM "public"."Epoch"  WHERE (("_root.or.tip.base"."epochNo") = ("number"))    LIMIT 1 ) AS "_root.or.tip.or.epoch.base"      ) AS "_root.or.tip.or.epoch" ON ('true')      ) AS "_root.or.tip" ON ('true')      ) AS "_root"
2024-07-13 05:03:19.548 UTC [18448] ERROR:  relation "public.TokenMint" does not exist at character 154
2024-07-13 05:03:19.548 UTC [18448] STATEMENT:  SELECT  json_build_object('aggregate', json_build_object('count', COUNT(*) ) ) AS "root" FROM  (SELECT  1  FROM  (SELECT DISTINCT ON ("assetId") *  FROM "public"."TokenMint"  WHERE ('true')     ) AS "_root.base"      ) AS "_root"
2024-07-13 05:03:19.548 UTC [18485] ERROR:  relation "public.Epoch" does not exist at character 288
2024-07-13 05:03:19.548 UTC [18485] STATEMENT:  SELECT  coalesce(json_agg("root" ORDER BY "root.pg.number" DESC NULLS FIRST), '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.base"."number" AS "number"       ) AS "_e"      ) ) AS "root", "_root.base"."number" AS "root.pg.number" FROM  (SELECT *  FROM "public"."Epoch"  WHERE ('true')   ORDER BY "number" DESC NULLS FIRST LIMIT 1 ) AS "_root.base"    ORDER BY "root.pg.number" DESC NULLS FIRST  ) AS "_root"
2024-07-13 05:03:19.548 UTC [18448] ERROR:  relation "public.Epoch" does not exist at character 296
2024-07-13 05:03:19.548 UTC [18448] STATEMENT:  SELECT  coalesce(json_agg("root" ORDER BY "root.pg.number" DESC NULLS FIRST), '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  "_root.or.adaPots"."adaPots" AS "adaPots"       ) AS "_e"      ) ) AS "root", "_root.base"."number" AS "root.pg.number" FROM(SELECT  *  FROM "public"."Epoch"  WHERE ('true')   ORDER BY "number" DESC NULLS FIRST LIMIT 1 ) AS "_root.base" LEFT OUTER JOIN LATERAL (SELECT  row_to_json((SELECT  "_e"  FROM  (SELECT  ("_root.or.adaPots.base"."reserves")::text AS "reserves"       ) AS "_e"      ) ) AS "adaPots" FROM  (SELECT  *  FROM "public"."AdaPots"WHERE (("_root.base"."number") = ("epochNo"))    LIMIT 1 ) AS "_root.or.adaPots.base"      ) AS "_root.or.adaPots" ON ('true')    ORDER BY "root.pg.number" DESC NULLS FIRST  ) AS "_root"

Cardano-graphql-hasura periodically outputs errors.

{
  "detail": {
    "http_info": {
      "content_encoding": null,
      "http_version": "HTTP/1.1",
      "ip": "192.168.80.3",
      "method": "POST",
      "status": 200,
      "url": "/v1/graphql"
    },
    "operation": {
      "error": {
        "code": "unexpected",
        "error": "database query error",
        "internal": {
          "arguments": [
            "(Oid 114, Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\", Binary))"
          ],
          "error": {
            "description": null,
            "exec_status": "FatalError",
            "hint": null,
            "message": "relation \"public.Cardano\" does not exist",
            "status_code": "42P01"
          },
          "prepared": true,
          "statement": "SELECT coalesce(json_agg(\"root\"), '[]') AS \"root\" FROM (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.or.tip\".\"tip\" AS \"tip\") AS \"_e\")) AS \"root\" FROM (SELECT * FROM \"public\".\"Cardano\" WHERE ('true') LIMIT 1) AS \"_root.base\" LEFT OUTER JOIN LATERAL (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.or.tip.base\".\"number\" AS \"number\", (\"_root.or.tip.base\".\"slotNo\")::text AS \"slotNo\", \"_root.or.tip.or.epoch\".\"epoch\" AS \"epoch\") AS \"_e\")) AS \"tip\" FROM (SELECT * FROM \"public\".\"Block\" WHERE ((\"_root.base\".\"tipBlockNo\") = (\"number\")) LIMIT 1) AS \"_root.or.tip.base\" LEFT OUTER JOIN LATERAL (SELECT row_to_json((SELECT \"_e\" FROM (SELECT \"_root.or.tip.or.epoch.base\".\"number\" AS \"number\") AS \"_e\")) AS \"epoch\" FROM (SELECT * FROM \"public\".\"Epoch\" WHERE ((\"_root.or.tip.base\".\"epochNo\") = (\"number\")) LIMIT 1) AS \"_root.or.tip.or.epoch.base\") AS \"_root.or.tip.or.epoch\" ON ('true')) AS \"_root.or.tip\" ON ('true')) AS \"_root\""
        },
        "path": "$"
      },
      "request_id": "809e92c7-e597-4a64-a810-663f4b7f07da",
      "request_mode": "error",
      "response_size": 1470,
      "uncompressed_response_size": 1470,
      "user_vars": {
        "x-hasura-role": "cardano-graphql"
      }
    },
    "request_id": "809e92c7-e597-4a64-a810-663f4b7f07da"
  },
  "level": "error",
  "timestamp": "2024-07-13T05:15:10.243+0000",
  "type": "http-log"
}

Cardano-graphql-server does not show any suspicious logs.

{"name":"cardano-graphql","hostname":"18848301b8a8","pid":1,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-07-13T05:04:48.595Z","v":0}
{"name":"cardano-graphql","hostname":"18848301b8a8","pid":1,"level":30,"module":"CardanoNodeClient","tip":{"slot":129280783,"id":"bf7dab8e1f9c2ac1844377ade3b276d711b3e3a7f922d65434110f79030aaf10"},"msg":"[Object: null prototype] {\n  slot: 129280783,\n  id: 'bf7dab8e1f9c2ac1844377ade3b276d711b3e3a7f922d65434110f79030aaf10'\n}","time":"2024-07-13T05:04:48.597Z","v":0}
{"name":"cardano-graphql","hostname":"18848301b8a8","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 100% | Asset: 0%","time":"2024-07-13T05:05:09.623Z","v":0}
{"name":"cardano-graphql","hostname":"18848301b8a8","pid":1,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-07-13T05:05:09.624Z","v":0}
{"name":"cardano-graphql","hostname":"18848301b8a8","pid":1,"level":30,"module":"CardanoNodeClient","tip":{"slot":129280813,"id":"21b13ea170775c00693d40e3599cf5f3ee8f35b0767bedeb9a515de212402cfd"},"msg":"[Object: null prototype] {\n  slot: 129280813,\n  id: '21b13ea170775c00693d40e3599cf5f3ee8f35b0767bedeb9a515de212402cfd'\n}","time":"2024-07-13T05:05:09.628Z","v":0}
{"name":"cardano-graphql","hostname":"18848301b8a8","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 100% | Asset: 0%","time":"2024-07-13T05:05:30.422Z","v":0}
{"name":"cardano-graphql","hostname":"18848301b8a8","pid":1,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-07-13T05:05:30.424Z","v":0}
{"name":"cardano-graphql","hostname":"18848301b8a8","pid":1,"level":30,"module":"CardanoNodeClient","tip":{"slot":129280831,"id":"5889091e0d6ca37cb7cab46cb6197f8a2f271de05f73c47031b4afb7618181b9"},"msg":"[Object: null prototype] {\n  slot: 129280831,\n  id: '5889091e0d6ca37cb7cab46cb6197f8a2f271de05f73c47031b4afb7618181b9'\n}","time":"2024-07-13T05:05:30.426Z","v":0}
{"name":"cardano-graphql","hostname":"18848301b8a8","pid":1,"level":30,"module":"CardanoNodeClient","msg":"initialized","time":"2024-07-13T05:07:59.729Z","v":0}
{"name":"cardano-graphql","hostname":"18848301b8a8","pid":1,"level":30,"module":"CardanoNodeClient","tip":{"slot":129280985,"id":"357191f5f1fee50d4a6ec830ddee7c18285c6b267e3920f8e8922d9ae85a68f8"},"msg":"[Object: null prototype] {\n  slot: 129280985,\n  id: '357191f5f1fee50d4a6ec830ddee7c18285c6b267e3920f8e8922d9ae85a68f8'\n}","time":"2024-07-13T05:07:59.737Z","v":0}
niso1985 commented 3 weeks ago

I assume that this is probably due to the token-metadata-registry not being properly activated. Since there is no problem now, I will close it.