search

cardano-foundation / cardano-graphql

GraphQL API for Cardano
Apache License 2.0
263 stars 104 forks source link

Cannot return null for non-nullable field Token.asset - Truncate does not help. #617

Closed bgiegel closed 3 years ago

bgiegel commented 3 years ago

Environment

cardano-graphql v5.1.0-mainnet cardano-node-ogmios-v4.0.0-mainnet cardano-db-sync:11.0.0

Platform

Platform version:

Runtime

Runtime version: node v10.24.0

Steps to reproduce the bug

I get an Cannot return null for non-nullable field Token.asset when executing this query

  blocks(where: { number: { _eq: 6298956 } }) {
    forgedAt
    transactions {
      fee
      deposit
      hash
      inputs {
        address
        sourceTxHash
        sourceTxIndex
        value
        tokens {
          asset {
            assetName
            policyId
          }
          quantity
        }
      }
      outputs {
        address
        index
        value
        tokens {
          asset {
            assetName
            policyId
          }
          quantity
        }
      }
    }
  }
}

I’ve already reported this error here :  #612. The resolution was to run a truncate on the PostgreSQL DB. I did it, it took a very long time to rebuild the table but then the error was gone. Now the thing is that the fix is working for a few hundreds blocks and then it comes back. I don’t know if this issue comes from graphql or db-sync or another component but it seems that something is messing with table regularly.

What is the expected behavior?

After a full rebuild of the table the query is working again permanently.

rhyslbw commented 3 years ago

cardano-node-v4.0.0-mainnet

Do you mean cardano-node-ogmios v4.1.0-mainnet ?

I don’t know if this issue comes from graphql or db-sync or another component but it seems that something is messing with table regularly.

The asset table is managed by cardano-graphql, so it should at least be isolated here. In the new version you can run the query I suggested here: https://github.com/input-output-hk/cardano-graphql/issues/612#issuecomment-930766064. Please run this and report results.

bgiegel commented 3 years ago

Do you mean cardano-node-ogmios v4.1.0-mainnet ?

yes sorry it’s cardano-node-ogmios-v4.0.0-mainnet.

I’ll make a try with the new versions. Thx.

bgiegel commented 3 years ago

Ok I have some news. I have setup a brand new ada installation on a dedicated vm. I just ran the docker-compose available on this repository with the latest version of the node and graphql. The sync just finished. And I still have the same error... At least now I can run your aggregate query : 

{
  "data": {
    "assets_aggregate": {
      "aggregate": {
        "count": "895295"
      }
    },
    "tokenMints_aggregate": {
      "aggregate": {
        "count": "1162536"
      }
    }
  }
}

Does this help ?

bgiegel commented 3 years ago

oh I get it it just means that the asset table wich is populated by the cardano graphql was not fully finished even though the node is fully synced with the network and the postgresql is synced with the node state.

Now running you query returns the same result for the asset than this tokensMints thing :

{
  "data": {
    "assets_aggregate": {
      "aggregate": {
        "count": "1163934"
      }
    },
    "tokenMints_aggregate": {
      "aggregate": {
        "count": "1163934"
      }
    }
  }
}

But if this assets table is populated by the graphql maybe our issue is that graphql is getting out of sync with the node right ?  Maybe we don’t put enough ressources on it.

rhyslbw commented 3 years ago

Correct. Comparing the aggregate sum of all unique tokenMints to the assets table indicates if the two chain-following services are in sync at the tip of the node.

Maybe we don’t put enough resources on it.

What does your stack look like?

bgiegel commented 3 years ago

What does your stack look like?

We created an helm chart based on the docker-composed provided in cardano-graphql and deployed everything on an Openshift 4 cluster.

At first we put some limits on the memory of the graphql container. Wich caused the container to be OOM killed. So I removed the limits. It stopped restarting. But the issue was still here. So I tried to restart the entire node sync from scratch with increase CPU request for the postgresql. And now it seems stable.

rhyslbw commented 3 years ago

Great, thanks for the detail. I'll close this for now, but we can reopen if needed.

alecalve commented 3 years ago

I am encountering a similar issue using the following versions:

cardano_fullnode_version: "1.30.1"
cardano_fullnode_db_sync_version: "11.0.4"
cardano_fullnode_db_explorer_api_version: "3.1.2"
cardano_fullnode_graphql_version: "6.0.0"
cardano_fullnode_ogmios_version: "v4.1.0-mainnet"

I get the following error when querying block 6360000:

Cannot return null for non-nullable field Token.transactionOutput

I tried to rebuild the assets table as mentionned in #612 but that didn't work too. The node has been upgraded several times (it's not a fresh start using these versions).

The schema I send is:

query GetBlockByHash($hash: Hash32Hex) {
    blocks(limit: 1, where: {hash: {_eq: $hash}}) {
        epochNo
        fees
        forgedAt
        slotLeader {
            description
            hash
        }
        hash
        number
        opCert
        slotInEpoch
        slotNo
        previousBlock { hash number previousBlock { hash } }
        protocolVersion
        size
        transactions {
            blockIndex
            collateral {
                address
                redeemer {
                    fee
                    index
                    purpose
                    scriptHash
                    transaction { hash }
                    unitMem
                    unitSteps
                }
                sourceTxHash
                sourceTxIndex
                tokens {
                    asset { assetId }
                    quantity
                    transactionOutput { transaction { hash }, index }
                }
                txHash
                value
            }
            deposit
            fee
            hash
            inputs {
                address
                redeemer {
                    fee
                    index
                    purpose
                    scriptHash
                    transaction { hash }
                    unitMem
                    unitSteps
                }
                sourceTxHash
                sourceTxIndex
                tokens {
                    asset { assetId }
                    quantity
                    transactionOutput { transaction { hash }, index }
                }
                txHash
                value
            }
            invalidBefore
            invalidHereafter
            metadata {
                key
                value
            }
            mint {
                asset { assetId }
                quantity
                transactionOutput { transaction { hash }, index }
            }
            outputs {
                address
                addressHasScript
                index
                txHash
                tokens {
                    asset { assetId }
                    quantity
                    transactionOutput { transaction { hash }, index }
                }
                value
            }
            redeemers {
                fee
                index
                purpose
                scriptHash
                transaction { hash }
                unitMem
                unitSteps
            }
            scripts {
                hash
                serialisedSize
                type
            }
            scriptSize
            size
            totalOutput
            includedAt
            validContract
            withdrawals {
                address
                amount
                redeemer {
                    fee
                    index
                    purpose
                    scriptHash
                    transaction { hash }
                    unitMem
                    unitSteps
                }
            }
        }
        transactionsCount
        vrfKey
    }
}
dmitri-algazin-copper commented 2 years ago

same issue on block 6566501: Cannot return null for non-nullable field Token.asset

CyberCyclone commented 2 years ago

@rhyslbw will a future version of cardano-graphql correct itself if it detects that Token.asset has gone out of sync? I just came across this on all 4 of my testnet servers (all have the assets and tokenMints out of sync). Due to the ever-growing size of the Cardano, I'm trying to find ways to avoid doing DB rebuilds.

CyberCyclone commented 2 years ago

I think this issue needs to be re-opened. On cardano-graphql 6.1.0 I've come across this after migrating. I restarted Docker thinking the migration was finished but now some of the TokenMint assets are missing and you can't query that TokenMint.

rcmorano commented 2 years ago

I'm facing lots of these issues lately and I do agree with @CyberCyclone, I think cardano-graphql should do some check/resync on its side whenever an asset is missing from the table. cc @rhyslbw

rcmorano commented 2 years ago

I've just observed that it will stop syncing assets when a db connection problem raises, and for whatever reason it has stopped for me around 4% of the sync and won't resume (not raising the exception anymore).

[0]

{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":30,"module":"HasuraClient","assetId":"5acc52d5696e52345aec108468050d9d743eb21d6e41305bbc23a27b4154484f4d","msg":"Adding metadata to asset","time":"2022-06-03T00:52:12.711Z","v":0}
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":20,"module":"DataFetcher","instance":"ServerHealth","value":{"startTime":"2022-05-27T08:49:05.529355378Z","lastKnownTip":{"slot":62630859,"hash":"f019d9ef9daaad5f110659b5653c0dbe4c8b6aeb1419dd08367d05d0afc49500","blockNo":7325468},"lastTipUpdate":"2022-06-02T19:12:31.503215526Z","networkSynchronization":0.99999,"currentEra":"Alonzo","metrics":{"sessionDurations":{"max":0,"min":0,"mean":135406113.08333334},"totalMessages":11510323,"runtimeStats":{"currentHeapSize":3156,"maxHeapSize":16330,"gcCpuTime":730487446667,"cpuTime":6582718759330},"activeConnections":3,"totalUnrouted":0,"totalConnections":15},"connectionStatus":"connected","currentEpoch":342,"slotInEpoch":250059},"msg":"value fetched after 30 seconds","time":"2022-06-03T00:52:34.406Z","v":0}
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":20,"module":"DataFetcher","instance":"ServerHealth","value":{"startTime":"2022-05-27T08:49:05.529355378Z","lastKnownTip":{"slot":62630859,"hash":"f019d9ef9daaad5f110659b5653c0dbe4c8b6aeb1419dd08367d05d0afc49500","blockNo":7325468},"lastTipUpdate":"2022-06-02T19:12:31.503215526Z","networkSynchronization":0.99999,"currentEra":"Alonzo","metrics":{"sessionDurations":{"max":0,"min":0,"mean":135406113.08333334},"totalMessages":11510323,"runtimeStats":{"currentHeapSize":3156,"maxHeapSize":16330,"gcCpuTime":730487446667,"cpuTime":6582718759330},"activeConnections":3,"totalUnrouted":0,"totalConnections":15},"connectionStatus":"connected","currentEpoch":342,"slotInEpoch":250059},"msg":"value fetched after 30 seconds","time":"2022-06-03T00:53:04.411Z","v":0}
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 99.96% | Asset: 4%","time":"2022-06-03T00:53:12.645Z","v":0}
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":20,"module":"CardanoNodeClient","slotNo":62651286,"msg":"getTipSlotNo","time":"2022-06-03T00:53:12.690Z","v":0}
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":30,"module":"HasuraClient","assetId":"2ad8fbdc6f18d97ce61b113301f05ff6b2a241c4e0bf5d06127d30a24f4343","msg":"Adding metadata to asset","time":"2022-06-03T00:53:18.644Z","v":0}
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":20,"module":"Worker","qty":500,"msg":"Processing jobs","time":"2022-06-03T00:53:27.470Z","v":0}
(node:1) UnhandledPromiseRejectionWarning: Error [ERR_UNHANDLED_ERROR]: Unhandled error. ({
  errno: -113,
  code: 'EHOSTUNREACH',
  syscall: 'connect',
  address: '10.43.103.166',
  port: 5432,
  queue: '__pgboss__maintenance',
  worker: 'f9a214d5-9653-44f7-8fe2-71497541ab9e'
})
    at new NodeError (internal/errors.js:322:7)
    at PgBoss.emit (events.js:389:17)
    at Manager.<anonymous> (/app/node_modules/pg-boss/src/index.js:86:37)
    at Manager.emit (events.js:400:28)
    at Worker.onError (/app/node_modules/pg-boss/src/manager.js:198:12)
    at Worker.start (/app/node_modules/pg-boss/src/worker.js:60:14)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 40)
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":20,"module":"DataFetcher","instance":"ServerHealth","value":{"startTime":"2022-05-27T08:49:05.529355378Z","lastKnownTip":{"slot":62630859,"hash":"f019d9ef9daaad5f110659b5653c0dbe4c8b6aeb1419dd08367d05d0afc49500","blockNo":7325468},"lastTipUpdate":"2022-06-02T19:12:31.503215526Z","networkSynchronization":0.99999,"currentEra":"Alonzo","metrics":{"sessionDurations":{"max":0,"min":0,"mean":135406113.08333334},"totalMessages":11510323,"runtimeStats":{"currentHeapSize":3156,"maxHeapSize":16330,"gcCpuTime":730487446667,"cpuTime":6582718759330},"activeConnections":3,"totalUnrouted":0,"totalConnections":15},"connectionStatus":"connected","currentEpoch":342,"slotInEpoch":250059},"msg":"value fetched after 30 seconds","time":"2022-06-03T00:53:34.413Z","v":0}
(node:1) UnhandledPromiseRejectionWarning: Error: connect EHOSTUNREACH 10.43.103.166:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 42)
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":30,"module":"Server","msg":"Sync Progress: cardano-db-sync: 99.96% | Asset: 4%","time":"2022-06-03T00:53:52.301Z","v":0}
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":20,"module":"CardanoNodeClient","slotNo":62651337,"msg":"getTipSlotNo","time":"2022-06-03T00:53:52.316Z","v":0}
(node:1) UnhandledPromiseRejectionWarning: Error: connect EHOSTUNREACH 10.43.103.166:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 43)
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":20,"module":"DataFetcher","instance":"ServerHealth","value":{"startTime":"2022-05-27T08:49:05.529355378Z","lastKnownTip":{"slot":62630859,"hash":"f019d9ef9daaad5f110659b5653c0dbe4c8b6aeb1419dd08367d05d0afc49500","blockNo":7325468},"lastTipUpdate":"2022-06-02T19:12:31.503215526Z","networkSynchronization":0.99999,"currentEra":"Alonzo","metrics":{"sessionDurations":{"max":0,"min":0,"mean":135406113.08333334},"totalMessages":11510323,"runtimeStats":{"currentHeapSize":3156,"maxHeapSize":16330,"gcCpuTime":730487446667,"cpuTime":6582718759330},"activeConnections":3,"totalUnrouted":0,"totalConnections":15},"connectionStatus":"connected","currentEpoch":342,"slotInEpoch":250059},"msg":"value fetched after 30 seconds","time":"2022-06-03T00:54:04.421Z","v":0}
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":20,"module":"DataFetcher","instance":"ServerHealth","value":{"startTime":"2022-05-27T08:49:05.529355378Z","lastKnownTip":{"slot":62630859,"hash":"f019d9ef9daaad5f110659b5653c0dbe4c8b6aeb1419dd08367d05d0afc49500","blockNo":7325468},"lastTipUpdate":"2022-06-02T19:12:31.503215526Z","networkSynchronization":0.99999,"currentEra":"Alonzo","metrics":{"sessionDurations":{"max":0,"min":0,"mean":135406113.08333334},"totalMessages":11510323,"runtimeStats":{"currentHeapSize":3156,"maxHeapSize":16330,"gcCpuTime":730487446667,"cpuTime":6582718759330},"activeConnections":3,"totalUnrouted":0,"totalConnections":15},"connectionStatus":"connected","currentEpoch":342,"slotInEpoch":250059},"msg":"value fetched after 30 seconds","time":"2022-06-03T00:54:34.424Z","v":0}
(node:1) UnhandledPromiseRejectionWarning: Error [ERR_UNHANDLED_ERROR]: Unhandled error. ({
  errno: -113,
  code: 'EHOSTUNREACH',
  syscall: 'connect',
  address: '10.43.103.166',
  port: 5432,
  queue: 'asset-metadata-fetch-update',
  worker: 'dbdc7e98-b44b-4afc-8b46-df1a5728198c'
})
    at new NodeError (internal/errors.js:322:7)
    at PgBoss.emit (events.js:389:17)
    at Manager.<anonymous> (/app/node_modules/pg-boss/src/index.js:86:37)
    at Manager.emit (events.js:400:28)
    at Worker.onError (/app/node_modules/pg-boss/src/manager.js:198:12)
    at Worker.start (/app/node_modules/pg-boss/src/worker.js:60:14)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 44)
{"name":"cardano-graphql","hostname":"cardano-graphql-95d5cd9fd-sjlwv","pid":1,"level":20,"module":"DataFetcher","instance":"ServerHealth","value":{"startTime":"2022-05-27T08:49:05.529355378Z","lastKnownTip":{"slot":62630859,"hash":"f019d9ef9daaad5f110659b5653c0dbe4c8b6aeb1419dd08367d05d0afc49500","blockNo":7325468},"lastTipUpdate":"2022-06-02T19:12:31.503215526Z","networkSynchronization":0.99999,"currentEra":"Alonzo","metrics":{"sessionDurations":{"max":0,"min":0,"mean":135406113.08333334},"totalMessages":11510323,"runtimeStats":{"currentHeapSize":3156,"maxHeapSize":16330,"gcCpuTime":730487446667,"cpuTime":6582718759330},"activeConnections":3,"totalUnrouted":0,"totalConnections":15},"connectionStatus":"connected","currentEpoch":342,"slotInEpoch":250059},"msg":"value fetched after 30 seconds","time":"2022-06-03T00:55:04.427Z","v":0}