Cardano completely stuck 9 Apr ~18UTC

[ksemaev]

Hi guys.

Cardano-http-bridge went in crash loop with

ERROR 2020-04-09T18:08:39Z: cardano_http_bridge::handlers::status: error while reading difficulty from HEAD: BlockNotFound([238, 139, 53, 179, 234, 113, 63, 87, 197, 0, 11, 148, 33, 120, 206, 248, 118, 209, 122, 144, 190, 68, 161, 131, 15, 155, 47, 14, 24, 123, 153, 20]) thread '' panicked at 'Box', cardano-deps/exe-common/src/sync.rs:46:21 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.

Can't create an issue in https://github.com/vacuumlabs/cardano-http-bridge so posting here :)

It results adalite-importer failing (I assume it can't connect) with error like:

{"name":"yoroi-importer","hostname":"adalite-importer-79fff755f9-4g4h4","pid":41,"level":50,"msg":"Scheduler.startAsync exited with an error: { Error: Request failed with status code 500\n at createError (/adalite/node_modules/axios/lib/core/createError.js:16:15)\n at settle (/adalite/node_modules/axios/lib/core/settle.js:18:12)\n at IncomingMessage.handleStreamEnd (/adalite/node_modules/axios/lib/adapters/http.js:201:11)\n at IncomingMessage.emit (events.js:198:15)\n at IncomingMessage.EventEmitter.emit (domain.js:481:20)\n at endReadableNT (_stream_readable.js:1139:12)\n at processTicksAndRejections (internal/process/task_queues.js:81:17)\n config:\n { adapter: [Function: httpAdapter],\n transformRequest: { '0': [Function: transformRequest] },\n transformResponse: { '0': [Function: transformResponse] },\n timeout: 0,\n xsrfCookieName: 'XSRF-TOKEN',\n xsrfHeaderName: 'X-XSRF-TOKEN',\n maxContentLength: -1,\n validateStatus: [Function: validateStatus],\n headers:\n { Accept: 'application/json, text/plain, /',\n 'User-Agent': 'axios/0.18.0' },\n method: 'get',\n url:\n 'http://cardano-http-bridge.coin-nodes.svc.cluster.local/mainnet/status',\n responseType: 'json',\n data: undefined },\n request:\n ClientRequest {\n _events:\n [Object: null prototype] {\n socket: [Function],\n abort: [Function],\n aborted: [Function],\n error: [Function],\n timeout: [Function],\n prefinish: [Function: requestOnPrefinish] },\n _eventsCount: 6,\n _maxListeners: undefined,\n outputData: [],\n outputSize: 0,\n writable: true,\n _last: true,\n chunkedEncoding: false,\n shouldKeepAlive: false,\n useChunkedEncodingByDefault: false,\n sendDate: false,\n _removedConnection: false,\n _removedContLen: false,\n _removedTE: false,\n _contentLength: 0,\n _hasBody: true,\n _trailer: '',\n finished: true,\n _headerSent: true,\n socket:\n Socket {\n connecting: false,\n _hadError: false,\n _parent: null,\n _host: 'cardano-http-bridge.coin-nodes.svc.cluster.local',\n _readableState: [ReadableState],\n readable: true,\n _events: [Object],\n _eventsCount: 7,\n _maxListeners: undefined,\n _writableState: [WritableState],\n writable: false,\n allowHalfOpen: false,\n _sockname: null,\n _pendingData: null,\n _pendingEncoding: '',\n server: null,\n _server: null,\n parser: null,\n _httpMessage: [Circular],\n [Symbol(asyncId)]: 24,\n [Symbol(kHandle)]: [TCP],\n [Symbol(lastWriteQueueSize)]: 0,\n [Symbol(timeout)]: null,\n [Symbol(kBytesRead)]: 0,\n [Symbol(kBytesWritten)]: 0 },\n connection:\n Socket {\n connecting: false,\n _hadError: false,\n _parent: null,\n _host: 'cardano-http-bridge.coin-nodes.svc.cluster.local',\n _readableState: [ReadableState],\n readable: true,\n _events: [Object],\n _eventsCount: 7,\n _maxListeners: undefined,\n _writableState: [WritableState],\n writable: false,\n allowHalfOpen: false,\n _sockname: null,\n _pendingData: null,\n _pendingEncoding: '',\n server: null,\n _server: null,\n parser: null,\n _httpMessage: [Circular],\n [Symbol(asyncId)]: 24,\n [Symbol(kHandle)]: [TCP],\n [Symbol(lastWriteQueueSize)]: 0,\n [Symbol(timeout)]: null,\n [Symbol(kBytesRead)]: 0,\n [Symbol(kBytesWritten)]: 0 },\n _header:\n 'GET /mainnet/status HTTP/1.1\r\nAccept: application/json, text/plain, /\r\nUser-Agent: axios/0.18.0\r\nHost: cardano-http-bridge.coin-nodes.svc.cluster.local\r\nConnection: close\r\n\r\n',\n _onPendingData: [Function: noopPendingOutput],\n agent:\n Agent {\n _events: [Object],\n _eventsCount: 1,\n _maxListeners: undefined,\n defaultPort: 80,\n protocol: 'http:',\n options: [Object],\n requests: {},\n sockets: [Object],\n freeSockets: {},\n keepAliveMsecs: 1000,\n keepAlive: false,\n maxSockets: Infinity,\n maxFreeSockets: 256 },\n socketPath: undefined,\n timeout: undefined,\n method: 'GET',\n path: '/mainnet/status',\n _ended: true,\n res:\n IncomingMessage {\n _readableState: [ReadableState],\n readable: false,\n _events: [Object],\n _eventsCount: 3,\n _maxListeners: undefined,\n socket: [Socket],\n connection: [Socket],\n httpVersionMajor: 1,\n httpVersionMinor: 1,\n httpVersion: '1.1',\n complete: true,\n headers: [Object],\n rawHeaders: [Array],\n trailers: {},\n rawTrailers: [],\n aborted: false,\n upgrade: false,\n url: '',\n method: null,\n statusCode: 500,\n statusMessage: 'Internal Server Error',\n client: [Socket],\n _consuming: false,\n _dumped: false,\n req: [Circular],\n responseUrl:\n 'http://cardano-http-bridge.coin-nodes.svc.cluster.local/mainnet/status',\n redirects: [] },\n aborted: false,\n timeoutCb: null,\n upgradeOrConnect: false,\n parser: null,\n maxHeadersCount: null,\n _redirectable:\n Writable {\n _writableState: [WritableState],\n writable: true,\n _events: [Object],\n _eventsCount: 2,\n _maxListeners: undefined,\n _options: [Object],\n _ended: true,\n _ending: true,\n _redirectCount: 0,\n _redirects: [],\n _requestBodyLength: 0,\n _requestBodyBuffers: [],\n _onNativeResponse: [Function],\n _currentRequest: [Circular],\n _currentUrl:\n 'http://cardano-http-bridge.coin-nodes.svc.cluster.local/mainnet/status' },\n [Symbol(isCorked)]: false,\n [Symbol(outHeadersKey)]:\n [Object: null prototype] { accept: [Array], 'user-agent': [Array], host: [Array] } },\n response:\n { status: 500,\n statusText: 'Internal Server Error',\n headers:\n { 'content-length': '0', date: 'Thu, 09 Apr 2020 18:25:06 GMT' },\n config:\n { adapter: [Function: httpAdapter],\n transformRequest: [Object],\n transformResponse: [Object],\n timeout: 0,\n xsrfCookieName: 'XSRF-TOKEN',\n xsrfHeaderName: 'X-XSRF-TOKEN',\n maxContentLength: -1,\n validateStatus: [Function: validateStatus],\n headers: [Object],\n method: 'get',\n url:\n 'http://cardano-http-bridge.coin-nodes.svc.cluster.local/mainnet/status',\n responseType: 'json',\n data: undefined },\n request:\n ClientRequest {\n _events: [Object],\n _eventsCount: 6,\n _maxListeners: undefined,\n outputData: [],\n outputSize: 0,\n writable: true,\n _last: true,\n chunkedEncoding: false,\n shouldKeepAlive: false,\n useChunkedEncodingByDefault: false,\n sendDate: false,\n _removedConnection: false,\n _removedContLen: false,\n _removedTE: false,\n _contentLength: 0,\n _hasBody: true,\n _trailer: '',\n finished: true,\n _headerSent: true,\n socket: [Socket],\n connection: [Socket],\n _header:\n 'GET /mainnet/status HTTP/1.1\r\nAccept: application/json, text/plain, /\r\nUser-Agent: axios/0.18.0\r\nHost: cardano-http-bridge.coin-nodes.svc.cluster.local\r\nConnection: close\r\n\r\n',\n _onPendingData: [Function: noopPendingOutput],\n agent: [Agent],\n socketPath: undefined,\n timeout: undefined,\n method: 'GET',\n path: '/mainnet/status',\n _ended: true,\n res: [IncomingMessage],\n aborted: false,\n timeoutCb: null,\n upgradeOrConnect: false,\n parser: null,\n maxHeadersCount: null,\n _redirectable: [Writable],\n [Symbol(isCorked)]: false,\n [Symbol(outHeadersKey)]: [Object] },\n data: '' } }","time":"2020-04-09T18:25:06.477Z","v":0}

Please advice

[refi93]

Seems like a fork has happened, which impacted also us and adalite-importer (the piece importing into SQL: https://github.com/vacuumlabs/adalite-importer) mishandled this exception. This is related cardano-http-bridge log on our side:

 INFO 2020-04-09T17:22:40Z: protocol::ntt: HANDSHAKE OK
INFO 2020-04-09T17:22:41Z: protocol::protocol: creating initial light connection 1024
INFO 2020-04-09T17:22:41Z: protocol::protocol: subscribing on light connection 1024
INFO 2020-04-09T17:23:33Z: protocol::protocol: new async light connection 67315 from node 53094fe718b43a0767
INFO 2020-04-09T17:23:34Z: protocol::protocol: received 1 asynchronous headers
INFO 2020-04-09T17:23:34Z: exe_common::sync: Network TIP is       : 9a7e65978d2b3ba1f8ebe821f625d1dc9a491d3d72ebeb6c40ff4050a0dba0d1 (185.16496) <- a9907005960490ffd2579697c3ed3d997af3cd9575714dc84dc83c4f50df8b9a
INFO 2020-04-09T17:23:34Z: exe_common::sync: Fetching from        : ee8b35b3ea713f57c5000b942178cef876d17a90be44a1830f9b2f0e187b9914 (185.16492)
INFO 2020-04-09T17:23:34Z: exe_common::sync: First unstable epoch : 185
WARN 2020-04-09T17:23:34Z: exe_common::sync: Detected fork: local tip is no longer part of the chain: BlockRef { hash: HeaderHash(Blake2b256(0xee8b35b3ea713f57c5000b942178cef876d17a90be44a1830f9b2f0e187b9914)), date: Normal(EpochSlotId { epoch: 185, slotid: 16492 }), parent: HeaderHash(Blake2b256(0xe31dce51cfc5bdb580723bcc2cb63f964a834331481cecc99bceb5b3b589ee26)) }
ERROR 2020-04-09T17:23:43Z: cardano_http_bridge::handlers::status: error while reading difficulty from HEAD: BlockNotFound([238, 139, 53, 179, 234, 113, 63, 87, 197, 0, 11, 148, 33, 120, 206, 248, 118, 209, 122, 144, 190, 68, 161, 131, 15, 155, 47, 14, 24, 123, 153, 20])
INFO 2020-04-09T17:26:20Z: protocol::protocol: new async light connection 67317 from node 5342b99e2fdd8653de
INFO 2020-04-09T17:27:35Z: protocol::protocol: new async light connection 67318 from node 53c4642afb31aa9635
INFO 2020-04-09T17:27:35Z: protocol::protocol: received 1 asynchronous headers
INFO 2020-04-09T17:27:35Z: exe_common::sync: Network TIP is       : 64a5cf5dd32cb748389975c19907b735a6321f12cc7b751e3a0c7149af0f22b0 (185.16508) <- 13415357b96bc045798bb9ea9350f1bd489ca229dc7ea85f046b106a58fe05f4

It has spontaneously recovered, but the adalite-importer part crashed, investigating. In our case it was enough to restart adalite-importer and it's working fine so far

[refi93]

I asked this question in tangata-manu repo as that's the repo adalite-importer is forked from: https://github.com/Emurgo/tangata-manu/issues/163

[refi93]

got reply from tangata-manu devs, apparently (at least our outage) has been related to having the adalite-importer docker image based on tangata-manu's master branch instead of master-shelley, i.e. it was outdated and likely lacking logic to handle such exceptions, we will update adalite-importer in the upcoming days, meanwhile a restart should fix it

[ksemaev]

@refi93 TY for investigation. In our case the cause was cardano-http-bridge getting stuck, so we just restored it from recent backup, and it successfully passed the fork.