Request failed with status code 404

[roccomuso]

Having this error and process crashes:

Scheduler.startAsync exited with an error: { Error: Request failed with status code 404\n    at createError (/adalite/node_modules/axios/lib/core/createError.js:16:15)\n    at settle (/adalite/node_modules/axios/lib/core/settle.js:18:12)\n    at IncomingMessage.handleStreamEnd (/adalite/node_modules/axios/lib/adapters/http.js:201:11)\n    at IncomingMessage.emit (events.js:198:15)\n    at IncomingMessage.EventEmitter.emit (domain.js:481:20)\n    at endReadableNT (_stream_readable.js:1139:12)\n    at processTicksAndRejections (internal/process/task_queues.js:81:17)\n  config:\n   { adapter: [Function: httpAdapter],\n     transformRequest: { '0': [Function: transformRequest] },\n     transformResponse: { '0': [Function: transformResponse] },\n     timeout: 0,\n     xsrfCookieName: 'XSRF-TOKEN',\n     xsrfHeaderName: 'X-XSRF-TOKEN',\n     maxContentLength: -1,\n     validateStatus: [Function: validateStatus],\n     headers:\n      { Accept: 'application/json, text/plain, */*',\n        'User-Agent': 'axios/0.18.0' },\n     method: 'get',\n     url:\n      'http://cardano-http-bridge.coin-nodes.svc.cluster.local/mainnet/height/3676830',\n     responseType: 'arraybuffer',\n     data: undefined },\n  request:\n   ClientRequest {\n     _events:\n      [Object: null prototype] {\n        socket: [Function],\n        abort: [Function],\n        aborted: [Function],\n        error: [Function],\n        timeout: [Function],\n        prefinish: [Function: requestOnPrefinish] },\n     _eventsCount: 6,\n     _maxListeners: undefined,\n     outputData: [],\n     outputSize: 0,\n     writable: true,\n     _last: true,\n     chunkedEncoding: false,\n     shouldKeepAlive: false,\n     useChunkedEncodingByDefault: false,\n     sendDate: false,\n     _removedConnection: false,\n     _removedContLen: false,\n     _removedTE: false,\n     _contentLength: 0,\n     _hasBody: true,\n     _trailer: '',\n     finished: true,\n     _headerSent: true,\n     socket:\n      Socket {\n        connecting: false,\n        _hadError: false,\n        _parent: null,\n        _host: 'cardano-http-bridge.coin-nodes.svc.cluster.local',\n        _readableState: [ReadableState],\n        readable: true,\n        _events: [Object],\n        _eventsCount: 7,\n        _maxListeners: undefined,\n        _writableState: [WritableState],\n        writable: false,\n        allowHalfOpen: false,\n        _sockname: null,\n        _pendingData: null,\n        _pendingEncoding: '',\n        server: null,\n        _server: null,\n        parser: null,\n        _httpMessage: [Circular],\n        [Symbol(asyncId)]: 143411,\n        [Symbol(kHandle)]: [TCP],\n        [Symbol(lastWriteQueueSize)]: 0,\n        [Symbol(timeout)]: null,\n        [Symbol(kBytesRead)]: 0,\n        [Symbol(kBytesWritten)]: 0 },\n     connection:\n      Socket {\n        connecting: false,\n        _hadError: false,\n        _parent: null,\n        _host: 'cardano-http-bridge.coin-nodes.svc.cluster.local',\n        _readableState: [ReadableState],\n        readable: true,\n        _events: [Object],\n        _eventsCount: 7,\n        _maxListeners: undefined,\n        _writableState: [WritableState],\n        writable: false,\n        allowHalfOpen: false,\n        _sockname: null,\n        _pendingData: null,\n        _pendingEncoding: '',\n        server: null,\n        _server: null,\n        parser: null,\n        _httpMessage: [Circular],\n        [Symbol(asyncId)]: 143411,\n        [Symbol(kHandle)]: [TCP],\n        [Symbol(lastWriteQueueSize)]: 0,\n        [Symbol(timeout)]: null,\n        [Symbol(kBytesRead)]: 0,\n        [Symbol(kBytesWritten)]: 0 },\n     _header:\n      'GET /mainnet/height/3676830 HTTP/1.1\\r\\nAccept: application/json, text/plain, */*\\r\\nUser-Agent: axios/0.18.0\\r\\nHost: cardano-http-bridge.coin-nodes.svc.cluster.local\\r\\nConnection: close\\r\\n\\r\\n',\n     _onPendingData: [Function: noopPendingOutput],\n     agent:\n      Agent {\n        _events: [Object],\n        _eventsCount: 1,\n        _maxListeners: undefined,\n        defaultPort: 80,\n        protocol: 'http:',\n        options: [Object],\n        requests: {},\n        sockets: [Object],\n        freeSockets: {},\n        keepAliveMsecs: 1000,\n        keepAlive: false,\n        maxSockets: Infinity,\n        maxFreeSockets: 256 },\n     socketPath: undefined,\n     timeout: undefined,\n     method: 'GET',\n     path: '/mainnet/height/3676830',\n     _ended: true,\n     res:\n      IncomingMessage {\n        _readableState: [ReadableState],\n        readable: false,\n        _events: [Object],\n        _eventsCount: 3,\n        _maxListeners: undefined,\n        socket: [Socket],\n        connection: [Socket],\n        httpVersionMajor: 1,\n        httpVersionMinor: 1,\n        httpVersion: '1.1',\n        complete: true,\n        headers: [Object],\n        rawHeaders: [Array],\n        trailers: {},\n        rawTrailers: [],\n        aborted: false,\n        upgrade: false,\n        url: '',\n        method: null,\n        statusCode: 404,\n        statusMessage: 'Not Found',\n        client: [Socket],\n        _consuming: false,\n        _dumped: false,\n        req: [Circular],\n        responseUrl:\n         'http://cardano-http-bridge.coin-nodes.svc.cluster.local/mainnet/height/3676830',\n        redirects: [] },\n     aborted: false,\n     timeoutCb: null,\n     upgradeOrConnect: false,\n     parser: null,\n     maxHeadersCount: null,\n     _redirectable:\n      Writable {\n        _writableState: [WritableState],\n        writable: true,\n        _events: [Object],\n        _eventsCount: 2,\n        _maxListeners: undefined,\n        _options: [Object],\n        _ended: true,\n        _ending: true,\n        _redirectCount: 0,\n        _redirects: [],\n        _requestBodyLength: 0,\n        _requestBodyBuffers: [],\n        _onNativeResponse: [Function],\n        _currentRequest: [Circular],\n        _currentUrl:\n         'http://cardano-http-bridge.coin-nodes.svc.cluster.local/mainnet/height/3676830' },\n     [Symbol(isCorked)]: false,\n     [Symbol(outHeadersKey)]:\n      [Object: null prototype] { accept: [Array], 'user-agent': [Array], host: [Array] } },\n  response:\n   { status: 404,\n     statusText: 'Not Found',\n     headers:\n      { 'content-length': '9',\n        'content-type': 'text/plain',\n        date: 'Thu, 23 Jan 2020 12:00:35 GMT' },\n     config:\n      { adapter: [Function: httpAdapter],\n        transformRequest: [Object],\n        transformResponse: [Object],\n        timeout: 0,\n        xsrfCookieName: 'XSRF-TOKEN',\n        xsrfHeaderName: 'X-XSRF-TOKEN',\n        maxContentLength: -1,\n        validateStatus: [Function: validateStatus],\n        headers: [Object],\n        method: 'get',\n        url:\n         'http://cardano-http-bridge.coin-nodes.svc.cluster.local/mainnet/height/3676830',\n        responseType: 'arraybuffer',\n        data: undefined },\n     request:\n      ClientRequest {\n        _events: [Object],\n        _eventsCount: 6,\n        _maxListeners: undefined,\n        outputData: [],\n        outputSize: 0,\n        writable: true,\n        _last: true,\n        chunkedEncoding: false,\n        shouldKeepAlive: false,\n        useChunkedEncodingByDefault: false,\n        sendDate: false,\n        _removedConnection: false,\n        _removedContLen: false,\n        _removedTE: false,\n        _contentLength: 0,\n        _hasBody: true,\n        _trailer: '',\n        finished: true,\n        _headerSent: true,\n        socket: [Socket],\n        connection: [Socket],\n        _header:\n         'GET /mainnet/height/3676830 HTTP/1.1\\r\\nAccept: application/json, text/plain, */*\\r\\nUser-Agent: axios/0.18.0\\r\\nHost: cardano-http-bridge.coin-nodes.svc.cluster.local\\r\\nConnection: close\\r\\n\\r\\n',\n        _onPendingData: [Function: noopPendingOutput],\n        agent: [Agent],\n        socketPath: undefined,\n        timeout: undefined,\n        method: 'GET',\n        path: '/mainnet/height/3676830',\n        _ended: true,\n        res: [IncomingMessage],\n        aborted: false,\n        timeoutCb: null,\n        upgradeOrConnect: false,\n        parser: null,\n        maxHeadersCount: null,\n        _redirectable: [Writable],\n        [Symbol(isCorked)]: false,\n        [Symbol(outHeadersKey)]: [Object] },\n     data: <Buffer 4e 6f 74 20 46 6f 75 6e 64> } }

[refi93]

Since in the logs are mentions of "cardano-http-bridge", I assume you are on the new-importer branch. My guess is that the error is caused by cardano-http-bridge being behind the state of the importer ("adalite-importer" service) - a reason for that may be that the storage of cardano-http-bridge was wiped.

But it's just a guess, more context would help - when does the error happen? Does it happen when you start docker-compose from scratch? Does it happen right after the start of the service or after some time has elaborated?

[roccomuso]

But why storage of cardano-http-bridge might be wiped? It happens after some time.

[refi93]

If you did not wipe it, then it shouldn't wipe by itself - does it start working again after restarting it?

[roccomuso]

yes it start working again after a restart

[refi93]

I rebuilt adalite-importer (dockerized fork of tangata-manu) based on latest commit in master branch of tangata manu: https://hub.docker.com/r/refi93/adalite-importer/builds

cardano-http-bridge docker image (https://hub.docker.com/r/refi93/cardano-http-bridge/builds) should be up to date with https://github.com/Emurgo/cardano-http-bridge

You can try pulling the new adalite-importer:latest image, restarting the instance and see if the problem persists after the update

[jaydp17]

I think we have found out why the importer is crashing with 404s.

Because we load balance between multiple instances of cardano-http-bridge, one of them fell behind & when the importer's request landed on the cardano-http-bridge which was behind it'd crash with a 404.

[refi93]

thanks for the insight @jaydp17 , closing the issue