search

cardano-foundation / cardano-graphql

GraphQL API for Cardano
Apache License 2.0
262 stars 104 forks source link

PreProd GraphQL API Fails To Start #758

Closed CyberCyclone closed 2 years ago

CyberCyclone commented 2 years ago

Summary

Starting up 'preprod' cardano-graphql using the below command. Mainnet has no issues and runs fine. Tested on 2 seperate servers.

export NETWORK=preprod &&\
docker pull inputoutput/cardano-graphql:7.0.0-${NETWORK} &&\
docker pull inputoutput/cardano-graphql-hasura:7.0.0 &&\
docker pull cardanosolutions/cardano-node-ogmios:v5.5.5_1.35.3-${NETWORK} &&\
API_PORT=3101 \
HASURA_PORT=8091 \
OGMIOS_PORT=1338 \
POSTGRES_PORT=5433 \
docker-compose -p ${NETWORK} up -d &&\
docker-compose -p ${NETWORK} logs -f

Prints the below error and I never hear from preprod-cardano-graphql-1 again after running for over an hour. Blocks are in-sync according to Hasura.

cardano-node-ogmios_1  | [cb7167b1:cardano.node.PeerSelectionCounters:Info:57] [2022-09-04 06:20:12.72 UTC] PeerSelectionCounters {coldPeers = 7, warmPeers = 23, hotPeers = 20}
cardano-node-ogmios_1  | {"severity":"Error","timestamp":"2022-09-04T06:20:13.648915666Z","thread":"29","message":{"WebSocket":{"exception":"MuxError MuxBearerClosed \"<socket: 19> closed when reading data, waiting on next header True\"","tag":"WebSocketUnknownException"}},"version":"v5.5.5"}
cardano-node-ogmios_1  | {"severity":"Info","timestamp":"2022-09-04T06:20:13.649000526Z","thread":"29","message":{"WebSocket":{"tag":"WebSocketConnectionEnded","userAgent":"User-Agent unknown"}},"version":"v5.5.5"}
cardano-node-ogmios_1  | {"severity":"Error","timestamp":"2022-09-04T06:20:13.649915666Z","thread":"54","message":{"WebSocket":{"exception":"MuxError MuxBearerClosed \"<socket: 22> closed when reading data, waiting on next header True\"","tag":"WebSocketUnknownException"}},"version":"v5.5.5"}
cardano-node-ogmios_1  | {"severity":"Info","timestamp":"2022-09-04T06:20:13.649956896Z","thread":"54","message":{"WebSocket":{"tag":"WebSocketConnectionEnded","userAgent":"User-Agent unknown"}},"version":"v5.5.5"}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"Server","msg":"Initializing","time":"2022-09-04T01:14:43.255Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"HasuraClient","msg":"Initializing","time":"2022-09-04T01:14:43.507Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":40,"module":"Db","msg":"pgSubscriber: cardano-db-sync-extended starting, schema will be reset","time":"2022-09-04T01:14:43.799Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"CardanoNodeClient","msg":"Initializing. This can take a few minutes...","time":"2022-09-04T01:14:44.291Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"CardanoNodeClient","msg":"Initialized","time":"2022-09-04T01:14:47.968Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"MetadataFetchClient","msg":"Initializing","time":"2022-09-04T01:14:47.969Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"MetadataFetchClient","msg":"Initialized","time":"2022-09-04T01:14:48.008Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"ChainFollower","msg":"Initializing","time":"2022-09-04T01:14:48.009Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"ChainFollower","msg":"Initialized","time":"2022-09-04T01:14:48.012Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"HasuraClient","msg":"Initialized","time":"2022-09-04T01:14:48.614Z","v":0}
preprod-cardano-graphql-1      | bunyan usage error: /app/node_modules/ws/lib/websocket.js:246: attempt to log with an unbound log method: `this` is: <ref *1> WebSocket {
preprod-cardano-graphql-1      |   _events: [Object: null prototype] {},
preprod-cardano-graphql-1      |   _eventsCount: 0,
preprod-cardano-graphql-1      |   _maxListeners: undefined,
preprod-cardano-graphql-1      |   _binaryType: 'nodebuffer',
preprod-cardano-graphql-1      |   _closeCode: 1006,
preprod-cardano-graphql-1      |   _closeFrameReceived: false,
preprod-cardano-graphql-1      |   _closeFrameSent: false,
preprod-cardano-graphql-1      |   _closeMessage: '',
preprod-cardano-graphql-1      |   _closeTimer: null,
preprod-cardano-graphql-1      |   _extensions: {},
preprod-cardano-graphql-1      |   _protocol: '',
preprod-cardano-graphql-1      |   _readyState: 3,
preprod-cardano-graphql-1      |   _receiver: Receiver {
preprod-cardano-graphql-1      |     _writableState: WritableState {
preprod-cardano-graphql-1      |       objectMode: false,
preprod-cardano-graphql-1      |       highWaterMark: 16384,
preprod-cardano-graphql-1      |       finalCalled: false,
preprod-cardano-graphql-1      |       needDrain: false,
preprod-cardano-graphql-1      |       ending: true,
preprod-cardano-graphql-1      |       ended: true,
preprod-cardano-graphql-1      |       finished: true,
preprod-cardano-graphql-1      |       destroyed: true,
preprod-cardano-graphql-1      |       decodeStrings: true,
preprod-cardano-graphql-1      |       defaultEncoding: 'utf8',
preprod-cardano-graphql-1      |       length: 0,
preprod-cardano-graphql-1      |       writing: false,
preprod-cardano-graphql-1      |       corked: 0,
preprod-cardano-graphql-1      |       sync: true,
preprod-cardano-graphql-1      |       bufferProcessing: false,
preprod-cardano-graphql-1      |       onwrite: [Function: bound onwrite],
preprod-cardano-graphql-1      |       writecb: null,
preprod-cardano-graphql-1      |       writelen: 0,
preprod-cardano-graphql-1      |       afterWriteTickInfo: null,
preprod-cardano-graphql-1      |       buffered: [],
preprod-cardano-graphql-1      |       bufferedIndex: 0,
preprod-cardano-graphql-1      |       allBuffers: true,
preprod-cardano-graphql-1      |       allNoop: true,
preprod-cardano-graphql-1      |       pendingcb: 0,
preprod-cardano-graphql-1      |       prefinished: true,
preprod-cardano-graphql-1      |       errorEmitted: false,
preprod-cardano-graphql-1      |       emitClose: true,
preprod-cardano-graphql-1      |       autoDestroy: true,
preprod-cardano-graphql-1      |       errored: null,
preprod-cardano-graphql-1      |       closed: true
preprod-cardano-graphql-1      |     },
preprod-cardano-graphql-1      |     _events: [Object: null prototype] {},
preprod-cardano-graphql-1      |     _eventsCount: 0,
preprod-cardano-graphql-1      |     _maxListeners: undefined,
preprod-cardano-graphql-1      |     _binaryType: 'nodebuffer',
preprod-cardano-graphql-1      |     _extensions: {},
preprod-cardano-graphql-1      |     _isServer: false,
preprod-cardano-graphql-1      |     _maxPayload: 134217728,
preprod-cardano-graphql-1      |     _bufferedBytes: 0,
preprod-cardano-graphql-1      |     _buffers: [],
preprod-cardano-graphql-1      |     _compressed: false,
preprod-cardano-graphql-1      |     _payloadLength: 0,
preprod-cardano-graphql-1      |     _mask: undefined,
preprod-cardano-graphql-1      |     _fragmented: 0,
preprod-cardano-graphql-1      |     _masked: false,
preprod-cardano-graphql-1      |     _fin: false,
preprod-cardano-graphql-1      |     _opcode: 0,
preprod-cardano-graphql-1      |     _totalPayloadLength: 0,
preprod-cardano-graphql-1      |     _messageLength: 0,
preprod-cardano-graphql-1      |     _fragments: [],
preprod-cardano-graphql-1      |     _state: 0,
preprod-cardano-graphql-1      |     _loop: false,
preprod-cardano-graphql-1      |     [Symbol(kCapture)]: false,
preprod-cardano-graphql-1      |     [Symbol(websocket)]: [Circular *1]
preprod-cardano-graphql-1      |   },
preprod-cardano-graphql-1      |   _sender: Sender {
preprod-cardano-graphql-1      |     _extensions: {},
preprod-cardano-graphql-1      |     _socket: Socket {
preprod-cardano-graphql-1      |       connecting: false,
preprod-cardano-graphql-1      |       _hadError: false,
preprod-cardano-graphql-1      |       _parent: null,
preprod-cardano-graphql-1      |       _host: 'cardano-node-ogmios',
preprod-cardano-graphql-1      |       _readableState: [ReadableState],
preprod-cardano-graphql-1      |       _events: [Object: null prototype],
preprod-cardano-graphql-1      |       _eventsCount: 2,
preprod-cardano-graphql-1      |       _maxListeners: undefined,
preprod-cardano-graphql-1      |       _writableState: [WritableState],
preprod-cardano-graphql-1      |       allowHalfOpen: false,
preprod-cardano-graphql-1      |       _sockname: null,
preprod-cardano-graphql-1      |       _pendingData: null,
preprod-cardano-graphql-1      |       _pendingEncoding: '',
preprod-cardano-graphql-1      |       server: null,
preprod-cardano-graphql-1      |       _server: null,
preprod-cardano-graphql-1      |       parser: null,
preprod-cardano-graphql-1      |       _httpMessage: null,
preprod-cardano-graphql-1      |       timeout: 0,
preprod-cardano-graphql-1      |       write: [Function: writeAfterFIN],
preprod-cardano-graphql-1      |       [Symbol(async_id_symbol)]: 400,
preprod-cardano-graphql-1      |       [Symbol(kHandle)]: null,
preprod-cardano-graphql-1      |       [Symbol(kSetNoDelay)]: true,
preprod-cardano-graphql-1      |       [Symbol(lastWriteQueueSize)]: 0,
preprod-cardano-graphql-1      |       [Symbol(timeout)]: null,
preprod-cardano-graphql-1      |       [Symbol(kBuffer)]: null,
preprod-cardano-graphql-1      |       [Symbol(kBufferCb)]: null,
preprod-cardano-graphql-1      |       [Symbol(kBufferGen)]: null,
preprod-cardano-graphql-1      |       [Symbol(kCapture)]: false,
preprod-cardano-graphql-1      |       [Symbol(kBytesRead)]: 138,
preprod-cardano-graphql-1      |       [Symbol(kBytesWritten)]: 233,
preprod-cardano-graphql-1      |       [Symbol(RequestTimeout)]: undefined,
preprod-cardano-graphql-1      |       [Symbol(websocket)]: undefined
preprod-cardano-graphql-1      |     },
preprod-cardano-graphql-1      |     _firstFragment: true,
preprod-cardano-graphql-1      |     _compress: false,
preprod-cardano-graphql-1      |     _bufferedBytes: 0,
preprod-cardano-graphql-1      |     _deflating: false,
preprod-cardano-graphql-1      |     _queue: []
preprod-cardano-graphql-1      |   },
preprod-cardano-graphql-1      |   _socket: Socket {
preprod-cardano-graphql-1      |     connecting: false,
preprod-cardano-graphql-1      |     _hadError: false,
preprod-cardano-graphql-1      |     _parent: null,
preprod-cardano-graphql-1      |     _host: 'cardano-node-ogmios',
preprod-cardano-graphql-1      |     _readableState: ReadableState {
preprod-cardano-graphql-1      |       objectMode: false,
preprod-cardano-graphql-1      |       highWaterMark: 16384,
preprod-cardano-graphql-1      |       buffer: BufferList { head: null, tail: null, length: 0 },
preprod-cardano-graphql-1      |       length: 0,
preprod-cardano-graphql-1      |       pipes: [],
preprod-cardano-graphql-1      |       flowing: true,
preprod-cardano-graphql-1      |       ended: true,
preprod-cardano-graphql-1      |       endEmitted: true,
preprod-cardano-graphql-1      |       reading: false,
preprod-cardano-graphql-1      |       sync: false,
preprod-cardano-graphql-1      |       needReadable: false,
preprod-cardano-graphql-1      |       emittedReadable: false,
preprod-cardano-graphql-1      |       readableListening: false,
preprod-cardano-graphql-1      |       resumeScheduled: false,
preprod-cardano-graphql-1      |       errorEmitted: false,
preprod-cardano-graphql-1      |       emitClose: false,
preprod-cardano-graphql-1      |       autoDestroy: false,
preprod-cardano-graphql-1      |       destroyed: true,
preprod-cardano-graphql-1      |       errored: null,
preprod-cardano-graphql-1      |       closed: true,
preprod-cardano-graphql-1      |       closeEmitted: true,
preprod-cardano-graphql-1      |       defaultEncoding: 'utf8',
preprod-cardano-graphql-1      |       awaitDrainWriters: null,
preprod-cardano-graphql-1      |       multiAwaitDrain: false,
preprod-cardano-graphql-1      |       readingMore: false,
preprod-cardano-graphql-1      |       dataEmitted: true,
preprod-cardano-graphql-1      |       decoder: null,
preprod-cardano-graphql-1      |       encoding: null,
preprod-cardano-graphql-1      |       [Symbol(kPaused)]: false
preprod-cardano-graphql-1      |     },
preprod-cardano-graphql-1      |     _events: [Object: null prototype] {
preprod-cardano-graphql-1      |       end: [Function: onReadableStreamEnd],
preprod-cardano-graphql-1      |       error: [Function: socketOnError]
preprod-cardano-graphql-1      |     },
preprod-cardano-graphql-1      |     _eventsCount: 2,
preprod-cardano-graphql-1      |     _maxListeners: undefined,
preprod-cardano-graphql-1      |     _writableState: WritableState {
preprod-cardano-graphql-1      |       objectMode: false,
preprod-cardano-graphql-1      |       highWaterMark: 16384,
preprod-cardano-graphql-1      |       finalCalled: true,
preprod-cardano-graphql-1      |       needDrain: false,
preprod-cardano-graphql-1      |       ending: true,
preprod-cardano-graphql-1      |       ended: true,
preprod-cardano-graphql-1      |       finished: true,
preprod-cardano-graphql-1      |       destroyed: true,
preprod-cardano-graphql-1      |       decodeStrings: false,
preprod-cardano-graphql-1      |       defaultEncoding: 'utf8',
preprod-cardano-graphql-1      |       length: 0,
preprod-cardano-graphql-1      |       writing: false,
preprod-cardano-graphql-1      |       corked: 0,
preprod-cardano-graphql-1      |       sync: false,
preprod-cardano-graphql-1      |       bufferProcessing: false,
preprod-cardano-graphql-1      |       onwrite: [Function: bound onwrite],
preprod-cardano-graphql-1      |       writecb: null,
preprod-cardano-graphql-1      |       writelen: 0,
preprod-cardano-graphql-1      |       afterWriteTickInfo: null,
preprod-cardano-graphql-1      |       buffered: [],
preprod-cardano-graphql-1      |       bufferedIndex: 0,
preprod-cardano-graphql-1      |       allBuffers: true,
preprod-cardano-graphql-1      |       allNoop: true,
preprod-cardano-graphql-1      |       pendingcb: 0,
preprod-cardano-graphql-1      |       prefinished: true,
preprod-cardano-graphql-1      |       errorEmitted: false,
preprod-cardano-graphql-1      |       emitClose: false,
preprod-cardano-graphql-1      |       autoDestroy: false,
preprod-cardano-graphql-1      |       errored: null,
preprod-cardano-graphql-1      |       closed: true,
preprod-cardano-graphql-1      |       closeEmitted: false,
preprod-cardano-graphql-1      |       writable: true
preprod-cardano-graphql-1      |     },
preprod-cardano-graphql-1      |     allowHalfOpen: false,
preprod-cardano-graphql-1      |     _sockname: null,
preprod-cardano-graphql-1      |     _pendingData: null,
preprod-cardano-graphql-1      |     _pendingEncoding: '',
preprod-cardano-graphql-1      |     server: null,
preprod-cardano-graphql-1      |     _server: null,
preprod-cardano-graphql-1      |     parser: null,
preprod-cardano-graphql-1      |     _httpMessage: null,
preprod-cardano-graphql-1      |     timeout: 0,
preprod-cardano-graphql-1      |     write: [Function: writeAfterFIN],
preprod-cardano-graphql-1      |     [Symbol(async_id_symbol)]: 400,
preprod-cardano-graphql-1      |     [Symbol(kHandle)]: null,
preprod-cardano-graphql-1      |     [Symbol(kSetNoDelay)]: true,
preprod-cardano-graphql-1      |     [Symbol(lastWriteQueueSize)]: 0,
preprod-cardano-graphql-1      |     [Symbol(timeout)]: null,
preprod-cardano-graphql-1      |     [Symbol(kBuffer)]: null,
preprod-cardano-graphql-1      |     [Symbol(kBufferCb)]: null,
preprod-cardano-graphql-1      |     [Symbol(kBufferGen)]: null,
preprod-cardano-graphql-1      |     [Symbol(kCapture)]: false,
preprod-cardano-graphql-1      |     [Symbol(kBytesRead)]: 138,
preprod-cardano-graphql-1      |     [Symbol(kBytesWritten)]: 233,
preprod-cardano-graphql-1      |     [Symbol(RequestTimeout)]: undefined,
preprod-cardano-graphql-1      |     [Symbol(websocket)]: undefined
preprod-cardano-graphql-1      |   },
preprod-cardano-graphql-1      |   _bufferedAmount: 0,
preprod-cardano-graphql-1      |   _isServer: false,
preprod-cardano-graphql-1      |   _redirects: 0,
preprod-cardano-graphql-1      |   _url: 'ws://cardano-node-ogmios:1337',
preprod-cardano-graphql-1      |   _req: null,
preprod-cardano-graphql-1      |   [Symbol(kCapture)]: false
preprod-cardano-graphql-1      | }
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":50,"module":"ChainFollower","code":1006,"msg":"","time":"2022-09-04T01:14:51.014Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"Worker","msg":"Starting","time":"2022-09-04T01:14:52.980Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"Worker","msg":"Starting","time":"2022-09-04T01:14:52.981Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"Worker","msg":"Started","time":"2022-09-04T01:14:52.982Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"ChainFollower","msg":"Starting","time":"2022-09-04T01:14:52.982Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"Worker","msg":"Started","time":"2022-09-04T01:14:53.069Z","v":0}
preprod-cardano-graphql-1      | {"name":"cardano-graphql","hostname":"32df7fb2f948","pid":1,"level":30,"module":"ChainFollower","msg":"Starting","time":"2022-09-04T01:14:53.069Z","v":0}

Steps to reproduce the bug

Startup using the below.

export NETWORK=preprod &&\
docker pull inputoutput/cardano-graphql:7.0.0-${NETWORK} &&\
docker pull inputoutput/cardano-graphql-hasura:7.0.0 &&\
docker pull cardanosolutions/cardano-node-ogmios:v5.5.5_1.35.3-${NETWORK} &&\
API_PORT=3101 \
HASURA_PORT=8091 \
OGMIOS_PORT=1338 \
POSTGRES_PORT=5433 \
docker-compose -p ${NETWORK} up -d &&\
docker-compose -p ${NETWORK} logs -f

Actual Result

  1. Can access the Hasura console and do queries against it.
  2. Can see that node is fully synced.
  3. Accessing api graphql console by 3101 gives "can't connect" error in the browser.

Expected Result

  1. Expected preprod-cardano-graphql-1 to be running and be able to access the console on port 3101.

Environment

Docker on Ubuntu 20.04.

Platform

Platform version

Ubuntu 20.04.4 LTS

Runtime

Runtime version

20.10.17

rhyslbw commented 2 years ago

Yeah looks like the p2p bug bit us once again :cry: We're looking to make a fix release tomorrow, but it's fairly simple to apply the resolution to your local branch to get it working.