graphql: TypeError: Cannot read property 'networkSynchronization' of undefined

mission-liao commented 1 year ago

Summary

This issue happens when submitTransaction with one transaction from one Shelly-address to one Byron-Address. It's 100% reproducible on my mainnet deployments, but is not reproducible on testnet.

I have several cardano-graphql mainnet deployments with latest version (8.0.0), all of them are reproducible.

Steps to reproduce the bug

No response

Actual Result

The call to submitTransaction API is failed with "graphql: TypeError: Cannot read property 'networkSynchronization' of undefined"

Expected Result

It should work

Environment

                      _
                     | |
  ____   ____   ____ | |
 / ___) / _  | / _  || |
( (___ ( ( | || | | || |
 \____) \_|| | \_|| ||_|
       (_____|    |_|
{
  os: {
    arch: 'x86_64',
    cpus: { count: 32, model: 'AMD EPYC 7501' },
    totalMem: 34359738368,
  },
  version: '8.0.0'
}

let me paste platform environment in later fields.

Platform

[X] Linux (Ubuntu)
[ ] Linux (Other)
[ ] macOS
[ ] Windows

Platform version

Ubuntu 20.04.4 LTS

Runtime

[ ] Node.js
[X] Docker

Runtime version

20.10.13

mission-liao commented 1 year ago

Here is the suspicious log from cardano-node-ogmios (I hide some potential privacy information with '**' or 'cc')

cardano-graphql-800-cardano-node-ogmios-1  | {"severity":"Info","timestamp":"2023-05-12T02:09:26.899396318Z","thread":"10222","message":{"WebSocket":{"contents":{"agency":"ClientAgency TokIdle","event":"send","tag":"SubmitTx","tx":"**************"},"tag":"WebSocketClient"}},"version":"v5.5.7"}
cardano-graphql-800-cardano-node-ogmios-1  |  [34m[7b1e2a1f:cardano.node.Mempool:Info:9641] [0m [2023-05-12 02:09:26.89 UTC] fromList [("err",Object (fromList [("kind",String "BabbageOutputTooSmall"),("outputs",Array [Array [Object (fromList [("address",String "********"),("datum",Null),("referenceScript",Null),("value",Object (fromList [("lovelace",Number 1000000.0),("policies",Object (fromList []))]))]),Number 1051640.0]])])),("kind",String "TraceMempoolRejectedTx"),("mempoolSize",Object (fromList [("bytes",Number 0.0),("numTxs",Number 0.0)])),("tx",Object (fromList [("txid",String "cccccccc")]))]
cardano-graphql-800-cardano-node-ogmios-1  | {"severity":"Error","timestamp":"2023-05-12T02:09:26.90019749Z","thread":"10208","message":{"WebSocket":{"exception":"DecoderFailure (LocalTxSubmission (GenTx (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (TPraos StandardCrypto) (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (ShelleyMAEra 'Mary StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (AlonzoEra StandardCrypto)) (': * (ShelleyBlock (Praos StandardCrypto) (BabbageEra StandardCrypto)) ('[] *))))))))) (HardForkApplyTxErr (': * ByronBlock (': * (ShelleyBlock (TPraos StandardCrypto) (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (ShelleyMAEra 'Mary StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (AlonzoEra StandardCrypto)) (': * (ShelleyBlock (Praos StandardCrypto) (BabbageEra StandardCrypto)) ('[] *))))))))) ServerAgency TokBusy) (DeserialiseFailure 12 \"expected list start\")","tag":"WebSocketUnknownException"}},"version":"v5.5.7"}
cardano-graphql-800-cardano-node-ogmios-1  | {"severity":"Info","timestamp":"2023-05-12T02:09:26.900253785Z","thread":"10208","message":{"WebSocket":{"tag":"WebSocketConnectionEnded","userAgent":"User-Agent unknown"}},"version":"v5.5.7"}
cardano-graphql-800-cardano-node-ogmios-1  | {"severity":"Info","timestamp":"2023-05-12T02:09:26.902999219Z","thread":"10276","message":{"WebSocket":{"mode":"FullSerialization","tag":"WebSocketConnectionAccepted","userAgent":"User-Agent unknown"}},"version":"v5.5.7"}
cardano-graphql-800-cardano-node-ogmios-1  | {"severity":"Info","timestamp":"2023-05-12T02:09:26.903142258Z","thread":"10280","message":{"WebSocket":{"contents":{"agency":"ClientAgency TokPropose","event":"send","tag":"ProposeVersions","versions":["NodeToClientV_10","NodeToClientV_11","NodeToClientV_12","NodeToClientV_13"]},"tag":"WebSocketClient"}},"version":"v5.5.7"}
cardano-graphql-800-cardano-node-ogmios-1  | {"severity":"Info","timestamp":"2023-05-12T02:09:26.903172059Z","thread":"10281","message":{"WebSocket":{"contents":{"agency":"ClientAgency TokPropose","event":"send","tag":"ProposeVersions","versions":["NodeToClientV_10","NodeToClientV_11","NodeToClientV_12","NodeToClientV_13"]},"tag":"WebSocketClient"}},"version":"v5.5.7"}
cardano-graphql-800-cardano-node-ogmios-1  | {"severity":"Info","timestamp":"2023-05-12T02:09:26.903269432Z","thread":"10280","message":{"WebSocket":{"contents":{"agency":"ServerAgency TokConfirm","event":"receive","tag":"AcceptVersion","version":"\"NodeToClientV_13\""},"tag":"WebSocketClient"}},"version":"v5.5.7"}
cardano-graphql-800-cardano-node-ogmios-1  | {"severity":"Info","timestamp":"2023-05-12T02:09:26.903289637Z","thread":"10281","message":{"WebSocket":{"contents":{"agency":"ServerAgency TokConfirm","event":"receive","tag":"AcceptVersion","version":"\"NodeToClientV_13\""},"tag":"WebSocketClient"}},"version":"v5.5.7"}

kingli-crypto commented 1 year ago

Did you managed to resolve the issue? We are running into similar problems with our graphql deployment

carlodx commented 9 months ago

Hi. We are also running into a similar issue, we're having a hard time figuring out the problem. Did you manage to find the cause of the problem? Thank you!

saravadeanil commented 3 months ago

Hi, I am facing the similar issue on preview network for cardano-graphql version == 8.0.3. Do we have a fix for this?

Kammerlo commented 3 months ago

I'll check it, thanks for reporting that this is still an issue!

Kammerlo commented 3 months ago

I tried to reproduce it. I think you need to help me, because I couldn't reproduce it. I used the latest version from main on Preprod.

What I did:

Build a transaction with the CardanoClientLib to send 1 Ada from a shelley Address to byron address
Submitted it with the following payload: { "query": "mutation { submitTransaction( transaction:\"TRANSACTION_HEX\") { hash } }" }
Got Response: { "data": { "submitTransaction": { "hash": "9451f579099900c6e31989556eb4d01854fee2e2c1ebfaa07df6eab0d56ead1e" } } }

And everything was fine: You can see the transaction which I submitted here: https://preprod.cardanoscan.io/transaction/9451f579099900c6e31989556eb4d01854fee2e2c1ebfaa07df6eab0d56ead1e

saravadeanil commented 3 months ago

Hi @Kammerlo, thanks for your reply.

I am running the following cardano releases on the preview network. Can you confirm if you are using the same releases?

cardano-node-ogmios = v6.2.0_8.9.0-preview
cardana-db-sync = v13.2.0.2
cardano-graphql-server =. v8.0.3-preview
cardano-graphql-background = v8.0.3-preview
postgres = v14.10-alpine
cardano-graphql-hasura = v8.0.3

And I am getting below error while performing rawTx with above on preview network.

Query

mutation submitTransaction($transaction: String!) {
    submitTransaction(transaction: $transaction) {
        hash
    }
}

Results

{
  "errors": [
    {
      "message": "TypeError: Cannot read properties of undefined (reading 'networkSynchronization')",
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ],
      "path": [
        "submitTransaction"
      ],
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR",
        "exception": {
          "stacktrace": [
            "Error: TypeError: Cannot read properties of undefined (reading 'networkSynchronization')",
            "    at /app/packages/api-cardano-db-hasura/dist/executableSchema.js:78:43",
            "    at Generator.throw (<anonymous>)",
            "    at rejected (/app/packages/api-cardano-db-hasura/dist/executableSchema.js:6:65)",
            "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          ]
        }
      }
    }
  ],
  "data": null
}

When I performed the same rawTx with below cardano releases it was completed successfully.

cardano-node = v8.0.0
ogmios = v5.5.7
cardana-db-sync =v13.1.1.3
cardano-graphql-server = v8.0.0
cardano-graphql-background = v8.0.0
postgres = v11.5-alpine
cardano-graphql-hasura = v8.0.0

Kammerlo commented 3 months ago

I found the potential error and will fix it today. @saravadeanil

Kammerlo commented 3 months ago

@saravadeanil I fixed it. It was an uncatched exception in the ogmios health checker, which lead to an undefined networkSynchronization object. All changes are in this PR. https://github.com/cardano-foundation/cardano-graphql/pull/874 Could you please check this branch out and test if it solves the issue for you as well? I tested it on different environments, but want to be sure it solves it for you as well.

saravadeanil commented 3 months ago

@Kammerlo Thanks a lot. For sure, I will test it out.

Just to confirm, I should build the cardano-graphql Docker image from the Dockerfile on your fixture branch?

Kammerlo commented 3 months ago

Yes, check out this branch and then docker compose up -d --build with your other used env vars. Thanks :)

Kammerlo commented 3 months ago

@saravadeanil any updates on this? Did it work for you?

saravadeanil commented 3 months ago

@Kammerlo Sorry for the delay. I was able to perform transaction. The issue is resolved. Thanks a lot 🎉

Kammerlo commented 3 months ago

Perfect :) Thanks for the response and I'm glad it worked! I will merge the branch soon.

saravadeanil commented 3 months ago

I am assuming you will be creating a new release for this fix? @Kammerlo

Kammerlo commented 2 months ago

I want to get two other PR's into main and then I will build the new release. This will be done hopefully this week.

cardano-foundation / cardano-graphql