search

medic / cht-sync

Data synchronization between CouchDB and PostgreSQL for the purpose of analytics.
GNU General Public License v3.0
4 stars 5 forks source link

CHT sync failing when multiple databases are set to be synced #165

Closed njuguna-n closed 4 weeks ago

njuguna-n commented 1 month ago

Describe the bug Adding multiple databases to be synced does not work as expected and throws an error (see logs below). Additionally the source column in the couchdb_progress table does not differentiate different databases being synced from the same instance

To Reproduce

  1. Update the COUCHDB_DBS value in the .env file to have multiple databases e.g. "medic medic-sentinel"
  2. Run the docker compose command to startup all containers
  3. Check the couch2pg container logs

Expected behavior Multiple databases are synced as expected and different tables from the same instance will be saved in separate rows in the couchdb_progress table.

Logs

2024-10-18 14:20:25 Node.js v20.18.0
2024-10-18 14:20:51 Downloading CouchDB changes feed from 0
2024-10-18 14:20:51 Error getting pending: AxiosError: Request failed with status code 404
2024-10-18 14:20:51     at settle (file:///node_modules/axios/lib/core/settle.js:19:12)
2024-10-18 14:20:51     at IncomingMessage.handleStreamEnd (file:///node_modules/axios/lib/adapters/http.js:589:11)
2024-10-18 14:20:51     at IncomingMessage.emit (node:events:531:35)
2024-10-18 14:20:51     at endReadableNT (node:internal/streams/readable:1696:12)
2024-10-18 14:20:51     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
2024-10-18 14:20:51     at Axios.request (file:///node_modules/axios/lib/core/Axios.js:45:41)
2024-10-18 14:20:51     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-10-18 14:20:51     at async getPending (file:///src/importer.js:164:15)
2024-10-18 14:20:51     at async importChangesBatch (file:///src/importer.js:139:15)
2024-10-18 14:20:51     at async default (file:///src/importer.js:179:17)
2024-10-18 14:20:51     at async default (file:///src/watcher.js:9:30) {
2024-10-18 14:20:51   code: 'ERR_BAD_REQUEST',
2024-10-18 14:20:51   config: {
2024-10-18 14:20:51     transitional: {
2024-10-18 14:20:51       silentJSONParsing: true,
2024-10-18 14:20:51       forcedJSONParsing: true,
2024-10-18 14:20:51       clarifyTimeoutError: false
2024-10-18 14:20:51     },
2024-10-18 14:20:51     adapter: [ 'xhr', 'http', 'fetch' ],
2024-10-18 14:20:51     transformRequest: [ [Function: transformRequest] ],
2024-10-18 14:20:51     transformResponse: [ [Function: transformResponse] ],
2024-10-18 14:20:51     timeout: 0,
2024-10-18 14:20:51     xsrfCookieName: 'XSRF-TOKEN',
2024-10-18 14:20:51     xsrfHeaderName: 'X-XSRF-TOKEN',
2024-10-18 14:20:51     maxContentLength: -1,
2024-10-18 14:20:51     maxBodyLength: -1,
2024-10-18 14:20:51     env: { FormData: [Function], Blob: [class Blob] },
2024-10-18 14:20:51     validateStatus: [Function: validateStatus],
2024-10-18 14:20:51     headers: Object [AxiosHeaders] {
2024-10-18 14:20:51       Accept: 'application/json, text/plain, */*',
2024-10-18 14:20:51       'Content-Type': undefined,
2024-10-18 14:20:51       'User-Agent': 'axios/1.7.2',
2024-10-18 14:20:51       'Accept-Encoding': 'gzip, compress, deflate, br'
2024-10-18 14:20:51     },
2024-10-18 14:20:51     method: 'get',
2024-10-18 14:20:51     url: 'http://medic:password@host.docker.internal:5984/medic medic-sentinel/_changes?limit=0&since=0',
2024-10-18 14:20:51     data: undefined
2024-10-18 14:20:51   },
2024-10-18 14:20:51   request: <ref *1> ClientRequest {
2024-10-18 14:20:51     _events: [Object: null prototype] {
2024-10-18 14:20:51       abort: [Function (anonymous)],
2024-10-18 14:20:51       aborted: [Function (anonymous)],
2024-10-18 14:20:51       connect: [Function (anonymous)],
2024-10-18 14:20:51       error: [Function (anonymous)],
2024-10-18 14:20:51       socket: [Function (anonymous)],
2024-10-18 14:20:51       timeout: [Function (anonymous)],
2024-10-18 14:20:51       finish: [Function: requestOnFinish]
2024-10-18 14:20:51     },
2024-10-18 14:20:51     _eventsCount: 7,
2024-10-18 14:20:51     _maxListeners: undefined,
2024-10-18 14:20:51     outputData: [],
2024-10-18 14:20:51     outputSize: 0,
2024-10-18 14:20:51     writable: true,
2024-10-18 14:20:51     destroyed: true,
2024-10-18 14:20:51     _last: true,
2024-10-18 14:20:51     chunkedEncoding: false,
2024-10-18 14:20:51     shouldKeepAlive: true,
2024-10-18 14:20:51     maxRequestsOnConnectionReached: false,
2024-10-18 14:20:51     _defaultKeepAlive: true,
2024-10-18 14:20:51     useChunkedEncodingByDefault: false,
2024-10-18 14:20:51     sendDate: false,
2024-10-18 14:20:51     _removedConnection: false,
2024-10-18 14:20:51     _removedContLen: false,
2024-10-18 14:20:51     _removedTE: false,
2024-10-18 14:20:51     strictContentLength: false,
2024-10-18 14:20:51     _contentLength: 0,
2024-10-18 14:20:51     _hasBody: true,
2024-10-18 14:20:51     _trailer: '',
2024-10-18 14:20:51     finished: true,
2024-10-18 14:20:51     _headerSent: true,
2024-10-18 14:20:51     _closed: true,
2024-10-18 14:20:51     socket: Socket {
2024-10-18 14:20:51       connecting: false,
2024-10-18 14:20:51       _hadError: false,
2024-10-18 14:20:51       _parent: null,
2024-10-18 14:20:51       _host: 'host.docker.internal',
2024-10-18 14:20:51       _closeAfterHandlingError: false,
2024-10-18 14:20:51       _events: [Object],
2024-10-18 14:20:51       _readableState: [ReadableState],
2024-10-18 14:20:51       _writableState: [WritableState],
2024-10-18 14:20:51       allowHalfOpen: false,
2024-10-18 14:20:51       _maxListeners: undefined,
2024-10-18 14:20:51       _eventsCount: 6,
2024-10-18 14:20:51       _sockname: null,
2024-10-18 14:20:51       _pendingData: null,
2024-10-18 14:20:51       _pendingEncoding: '',
2024-10-18 14:20:51       server: null,
2024-10-18 14:20:51       _server: null,
2024-10-18 14:20:51       timeout: 5000,
2024-10-18 14:20:51       parser: null,
2024-10-18 14:20:51       _httpMessage: null,
2024-10-18 14:20:51       [Symbol(async_id_symbol)]: -1,
2024-10-18 14:20:51       [Symbol(kHandle)]: [TCP],
2024-10-18 14:20:51       [Symbol(lastWriteQueueSize)]: 0,
2024-10-18 14:20:51       [Symbol(timeout)]: Timeout {
2024-10-18 14:20:51         _idleTimeout: 5000,
2024-10-18 14:20:51         _idlePrev: [TimersList],
2024-10-18 14:20:51         _idleNext: [TimersList],
2024-10-18 14:20:51         _idleStart: 510,
2024-10-18 14:20:51         _onTimeout: [Function: bound ],
2024-10-18 14:20:51         _timerArgs: undefined,
2024-10-18 14:20:51         _repeat: null,
2024-10-18 14:20:51         _destroyed: false,
2024-10-18 14:20:51         [Symbol(refed)]: false,
2024-10-18 14:20:51         [Symbol(kHasPrimitive)]: false,
2024-10-18 14:20:51         [Symbol(asyncId)]: 371,
2024-10-18 14:20:51         [Symbol(triggerId)]: 369
2024-10-18 14:20:51       },
2024-10-18 14:20:51       [Symbol(kBuffer)]: null,
2024-10-18 14:20:51       [Symbol(kBufferCb)]: null,
2024-10-18 14:20:51       [Symbol(kBufferGen)]: null,
2024-10-18 14:20:51       [Symbol(shapeMode)]: true,
2024-10-18 14:20:51       [Symbol(kCapture)]: false,
2024-10-18 14:20:51       [Symbol(kSetNoDelay)]: true,
2024-10-18 14:20:51       [Symbol(kSetKeepAlive)]: true,
2024-10-18 14:20:51       [Symbol(kSetKeepAliveInitialDelay)]: 1,
2024-10-18 14:20:51       [Symbol(kBytesRead)]: 0,
2024-10-18 14:20:51       [Symbol(kBytesWritten)]: 0
2024-10-18 14:20:51     },
2024-10-18 14:20:51     _header: 'GET /medic%20medic-sentinel/_changes?limit=0&since=0 HTTP/1.1\r\n' +
2024-10-18 14:20:51       'Accept: application/json, text/plain, */*\r\n' +
2024-10-18 14:20:51       'User-Agent: axios/1.7.2\r\n' +
2024-10-18 14:20:51       'Accept-Encoding: gzip, compress, deflate, br\r\n' +
2024-10-18 14:20:51       'Host: host.docker.internal:5984\r\n' +
2024-10-18 14:20:51       'Authorization: Basic bWVkaWM6cGFzc3dvcmQ=\r\n' +
2024-10-18 14:20:51       'Connection: keep-alive\r\n' +
2024-10-18 14:20:51       '\r\n',
2024-10-18 14:20:51     _keepAliveTimeout: 0,
2024-10-18 14:20:51     _onPendingData: [Function: nop],
2024-10-18 14:20:51     agent: Agent {
2024-10-18 14:20:51       _events: [Object: null prototype],
2024-10-18 14:20:51       _eventsCount: 2,
2024-10-18 14:20:51       _maxListeners: undefined,
2024-10-18 14:20:51       defaultPort: 80,
2024-10-18 14:20:51       protocol: 'http:',
2024-10-18 14:20:51       options: [Object: null prototype],
2024-10-18 14:20:51       requests: [Object: null prototype] {},
2024-10-18 14:20:51       sockets: [Object: null prototype] {},
2024-10-18 14:20:51       freeSockets: [Object: null prototype],
2024-10-18 14:20:51       keepAliveMsecs: 1000,
2024-10-18 14:20:51       keepAlive: true,
2024-10-18 14:20:51       maxSockets: Infinity,
2024-10-18 14:20:51       maxFreeSockets: 256,
2024-10-18 14:20:51       scheduling: 'lifo',
2024-10-18 14:20:51       maxTotalSockets: Infinity,
2024-10-18 14:20:51       totalSocketCount: 1,
2024-10-18 14:20:51       [Symbol(shapeMode)]: false,
2024-10-18 14:20:51       [Symbol(kCapture)]: false
2024-10-18 14:20:51     },
2024-10-18 14:20:51     socketPath: undefined,
2024-10-18 14:20:51     method: 'GET',
2024-10-18 14:20:51     maxHeaderSize: undefined,
2024-10-18 14:20:51     insecureHTTPParser: undefined,
2024-10-18 14:20:51     joinDuplicateHeaders: undefined,
2024-10-18 14:20:51     path: '/medic%20medic-sentinel/_changes?limit=0&since=0',
2024-10-18 14:20:51     _ended: true,
2024-10-18 14:20:51     res: IncomingMessage {
2024-10-18 14:20:51       _events: [Object],
2024-10-18 14:20:51       _readableState: [ReadableState],
2024-10-18 14:20:51       _maxListeners: undefined,
2024-10-18 14:20:51       socket: null,
2024-10-18 14:20:51       httpVersionMajor: 1,
2024-10-18 14:20:51       httpVersionMinor: 1,
2024-10-18 14:20:51       httpVersion: '1.1',
2024-10-18 14:20:51       complete: true,
2024-10-18 14:20:51       rawHeaders: [Array],
2024-10-18 14:20:51       rawTrailers: [],
2024-10-18 14:20:51       joinDuplicateHeaders: undefined,
2024-10-18 14:20:51       aborted: false,
2024-10-18 14:20:51       upgrade: false,
2024-10-18 14:20:51       url: '',
2024-10-18 14:20:51       method: null,
2024-10-18 14:20:51       statusCode: 404,
2024-10-18 14:20:51       statusMessage: 'Object Not Found',
2024-10-18 14:20:51       client: [Socket],
2024-10-18 14:20:51       _consuming: false,
2024-10-18 14:20:51       _dumped: false,
2024-10-18 14:20:51       req: [Circular *1],
2024-10-18 14:20:51       _eventsCount: 4,
2024-10-18 14:20:51       responseUrl: 'http://medic:password@host.docker.internal:5984/medic%20medic-sentinel/_changes?limit=0&since=0',
2024-10-18 14:20:51       redirects: [],
2024-10-18 14:20:51       [Symbol(shapeMode)]: true,
2024-10-18 14:20:51       [Symbol(kCapture)]: false,
2024-10-18 14:20:51       [Symbol(kHeaders)]: [Object],
2024-10-18 14:20:51       [Symbol(kHeadersCount)]: 14,
2024-10-18 14:20:51       [Symbol(kTrailers)]: null,
2024-10-18 14:20:51       [Symbol(kTrailersCount)]: 0
2024-10-18 14:20:51     },
2024-10-18 14:20:51     aborted: false,
2024-10-18 14:20:51     timeoutCb: null,
2024-10-18 14:20:51     upgradeOrConnect: false,
2024-10-18 14:20:51     parser: null,
2024-10-18 14:20:51     maxHeadersCount: null,
2024-10-18 14:20:51     reusedSocket: false,
2024-10-18 14:20:51     host: 'host.docker.internal',
2024-10-18 14:20:51     protocol: 'http:',
2024-10-18 14:20:51     _redirectable: Writable {
2024-10-18 14:20:51       _events: [Object],
2024-10-18 14:20:51       _writableState: [WritableState],
2024-10-18 14:20:51       _maxListeners: undefined,
2024-10-18 14:20:51       _options: [Object],
2024-10-18 14:20:51       _ended: true,
2024-10-18 14:20:51       _ending: true,
2024-10-18 14:20:51       _redirectCount: 0,
2024-10-18 14:20:51       _redirects: [],
2024-10-18 14:20:51       _requestBodyLength: 0,
2024-10-18 14:20:51       _requestBodyBuffers: [],
2024-10-18 14:20:51       _eventsCount: 3,
2024-10-18 14:20:51       _onNativeResponse: [Function (anonymous)],
2024-10-18 14:20:51       _currentRequest: [Circular *1],
2024-10-18 14:20:51       _currentUrl: 'http://medic:password@host.docker.internal:5984/medic%20medic-sentinel/_changes?limit=0&since=0',
2024-10-18 14:20:51       [Symbol(shapeMode)]: true,
2024-10-18 14:20:51       [Symbol(kCapture)]: false
2024-10-18 14:20:51     },
2024-10-18 14:20:51     [Symbol(shapeMode)]: false,
2024-10-18 14:20:51     [Symbol(kCapture)]: false,
2024-10-18 14:20:51     [Symbol(kBytesWritten)]: 0,
2024-10-18 14:20:51     [Symbol(kNeedDrain)]: false,
2024-10-18 14:20:51     [Symbol(corked)]: 0,
2024-10-18 14:20:51     [Symbol(kOutHeaders)]: [Object: null prototype] {
2024-10-18 14:20:51       accept: [Array],
2024-10-18 14:20:51       'user-agent': [Array],
2024-10-18 14:20:51       'accept-encoding': [Array],
2024-10-18 14:20:51       host: [Array],
2024-10-18 14:20:51       authorization: [Array]
2024-10-18 14:20:51     },
2024-10-18 14:20:51     [Symbol(errored)]: null,
2024-10-18 14:20:51     [Symbol(kHighWaterMark)]: 16384,
2024-10-18 14:20:51     [Symbol(kRejectNonStandardBodyWrites)]: false,
2024-10-18 14:20:51     [Symbol(kUniqueHeaders)]: null
2024-10-18 14:20:51   },
2024-10-18 14:20:51   response: {
2024-10-18 14:20:51     status: 404,
2024-10-18 14:20:51     statusText: 'Object Not Found',
2024-10-18 14:20:51     headers: Object [AxiosHeaders] {
2024-10-18 14:20:51       'cache-control': 'must-revalidate',
2024-10-18 14:20:51       'content-length': '58',
2024-10-18 14:20:51       'content-type': 'application/json',
2024-10-18 14:20:51       date: 'Fri, 18 Oct 2024 11:20:51 GMT',
2024-10-18 14:20:51       server: 'CouchDB/3.3.3 (Erlang OTP/24)',
2024-10-18 14:20:51       'x-couch-request-id': '1158739112',
2024-10-18 14:20:51       'x-couchdb-body-time': '0'
2024-10-18 14:20:51     },
2024-10-18 14:20:51     config: {
2024-10-18 14:20:51       transitional: [Object],
2024-10-18 14:20:51       adapter: [Array],
2024-10-18 14:20:51       transformRequest: [Array],
2024-10-18 14:20:51       transformResponse: [Array],
2024-10-18 14:20:51       timeout: 0,
2024-10-18 14:20:51       xsrfCookieName: 'XSRF-TOKEN',
2024-10-18 14:20:51       xsrfHeaderName: 'X-XSRF-TOKEN',
2024-10-18 14:20:51       maxContentLength: -1,
2024-10-18 14:20:51       maxBodyLength: -1,
2024-10-18 14:20:51       env: [Object],
2024-10-18 14:20:51       validateStatus: [Function: validateStatus],
2024-10-18 14:20:51       headers: [Object [AxiosHeaders]],
2024-10-18 14:20:51       method: 'get',
2024-10-18 14:20:51       url: 'http://medic:password@host.docker.internal:5984/medic medic-sentinel/_changes?limit=0&since=0',
2024-10-18 14:20:51       data: undefined
2024-10-18 14:20:51     },
2024-10-18 14:20:51     request: <ref *1> ClientRequest {
2024-10-18 14:20:51       _events: [Object: null prototype],
2024-10-18 14:20:51       _eventsCount: 7,
2024-10-18 14:20:51       _maxListeners: undefined,
2024-10-18 14:20:51       outputData: [],
2024-10-18 14:20:51       outputSize: 0,
2024-10-18 14:20:51       writable: true,
2024-10-18 14:20:51       destroyed: true,
2024-10-18 14:20:51       _last: true,
2024-10-18 14:20:51       chunkedEncoding: false,
2024-10-18 14:20:51       shouldKeepAlive: true,
2024-10-18 14:20:51       maxRequestsOnConnectionReached: false,
2024-10-18 14:20:51       _defaultKeepAlive: true,
2024-10-18 14:20:51       useChunkedEncodingByDefault: false,
2024-10-18 14:20:51       sendDate: false,
2024-10-18 14:20:51       _removedConnection: false,
2024-10-18 14:20:51       _removedContLen: false,
2024-10-18 14:20:51       _removedTE: false,
2024-10-18 14:20:51       strictContentLength: false,
2024-10-18 14:20:51       _contentLength: 0,
2024-10-18 14:20:51       _hasBody: true,
2024-10-18 14:20:51       _trailer: '',
2024-10-18 14:20:51       finished: true,
2024-10-18 14:20:51       _headerSent: true,
2024-10-18 14:20:51       _closed: true,
2024-10-18 14:20:51       socket: [Socket],
2024-10-18 14:20:51       _header: 'GET /medic%20medic-sentinel/_changes?limit=0&since=0 HTTP/1.1\r\n' +
2024-10-18 14:20:51         'Accept: application/json, text/plain, */*\r\n' +
2024-10-18 14:20:51         'User-Agent: axios/1.7.2\r\n' +
2024-10-18 14:20:51         'Accept-Encoding: gzip, compress, deflate, br\r\n' +
2024-10-18 14:20:51         'Host: host.docker.internal:5984\r\n' +
2024-10-18 14:20:51         'Authorization: Basic bWVkaWM6cGFzc3dvcmQ=\r\n' +
2024-10-18 14:20:51         'Connection: keep-alive\r\n' +
2024-10-18 14:20:51         '\r\n',
2024-10-18 14:20:51       _keepAliveTimeout: 0,
2024-10-18 14:20:51       _onPendingData: [Function: nop],
2024-10-18 14:20:51       agent: [Agent],
2024-10-18 14:20:51       socketPath: undefined,
2024-10-18 14:20:51       method: 'GET',
2024-10-18 14:20:51       maxHeaderSize: undefined,
2024-10-18 14:20:51       insecureHTTPParser: undefined,
2024-10-18 14:20:51       joinDuplicateHeaders: undefined,
2024-10-18 14:20:51       path: '/medic%20medic-sentinel/_changes?limit=0&since=0',
2024-10-18 14:20:51       _ended: true,
2024-10-18 14:20:51       res: [IncomingMessage],
2024-10-18 14:20:51       aborted: false,
2024-10-18 14:20:51       timeoutCb: null,
2024-10-18 14:20:51       upgradeOrConnect: false,
2024-10-18 14:20:51       parser: null,
2024-10-18 14:20:51       maxHeadersCount: null,
2024-10-18 14:20:51       reusedSocket: false,
2024-10-18 14:20:51       host: 'host.docker.internal',
2024-10-18 14:20:51       protocol: 'http:',
2024-10-18 14:20:51       _redirectable: [Writable],
2024-10-18 14:20:51       [Symbol(shapeMode)]: false,
2024-10-18 14:20:51       [Symbol(kCapture)]: false,
2024-10-18 14:20:51       [Symbol(kBytesWritten)]: 0,
2024-10-18 14:20:51       [Symbol(kNeedDrain)]: false,
2024-10-18 14:20:51       [Symbol(corked)]: 0,
2024-10-18 14:20:51       [Symbol(kOutHeaders)]: [Object: null prototype],
2024-10-18 14:20:51       [Symbol(errored)]: null,
2024-10-18 14:20:51       [Symbol(kHighWaterMark)]: 16384,
2024-10-18 14:20:51       [Symbol(kRejectNonStandardBodyWrites)]: false,
2024-10-18 14:20:51       [Symbol(kUniqueHeaders)]: null
2024-10-18 14:20:51     },
2024-10-18 14:20:51     data: { error: 'not_found', reason: 'Database does not exist.' }
2024-10-18 14:20:51   }
2024-10-18 14:20:51 }
2024-10-18 14:20:51 
2024-10-18 14:20:51 node:internal/process/promises:391
2024-10-18 14:20:51     triggerUncaughtException(err, true /* fromPromise */);
2024-10-18 14:20:51     ^
2024-10-18 14:20:51 {
2024-10-18 14:20:51   error: 'not_found',
2024-10-18 14:20:51   reason: 'Database does not exist.',
2024-10-18 14:20:51   status: 404,
2024-10-18 14:20:51   name: 'not_found',
2024-10-18 14:20:51   message: 'Database does not exist.',
2024-10-18 14:20:51   stack: 'Error\n' +
2024-10-18 14:20:51     '    at Object.generateErrorFromResponse (/node_modules/pouchdb-errors/lib/index.js:100:18)\n' +
2024-10-18 14:20:51     '    at fetchJSON (/node_modules/pouchdb-adapter-http/lib/index.js:251:31)\n' +
2024-10-18 14:20:51     '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n' +
2024-10-18 14:20:51     '    at async fetchData (/node_modules/pouchdb-adapter-http/lib/index.js:1029:24)'
2024-10-18 14:20:51 }

Screenshots

Environment

Additional context

medic-ci commented 4 weeks ago

:tada: This issue has been resolved in version 1.1.4 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket: