search

fititnt / uwazi-docker

Dockerized version of Uwazi (“openness" in Swahili). HURIDOCS designed Uwazi to make human rights information more open and accessible to the defenders who need it.
The Unlicense
11 stars 4 forks source link

redis connection in broken state #50

Closed mayeulk closed 12 months ago

mayeulk commented 1 year ago

After a while (maybe 30 or 60 minutes?), the uwazi-docker_uwazi_1 container stops. Below are extracts of the logs. Attached are longer logs. Happy to provide more if needed.

2023-07-25T20:38:37.944Z [uwazi_development] uncaughtException: Redis connection in broken state: connection timeout exceeded. Error: Redis connection in broken state: connection timeout exceeded. at RedisClient.connection_gone (/home/node/uwazi/prod/node_modules/redis/index.js:588:19) at RedisClient.on_error (/home/node/uwazi/prod/node_modules/redis/index.js:346:10) at Socket. (/home/node/uwazi/prod/node_modules/redis/index.js:223:14) at Socket.emit (node:events:513:28) at Socket.emit (node:domain:489:12) at emitErrorNT (node:internal/streams/destroy:151:8) at emitErrorCloseNT (node:internal/streams/destroy:116:3) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) [Tenant error] Error: Accessing nonexistent async context 2023-07-25T20:38:37.946Z [uwazi_development] uncaught exception or unhandled rejection, Node process finished !! Error: Redis connection in broken state: connection timeout exceeded. at RedisClient.connection_gone (/home/node/uwazi/prod/node_modules/redis/index.js:588:19) at RedisClient.on_error (/home/node/uwazi/prod/node_modules/redis/index.js:346:10) at Socket. (/home/node/uwazi/prod/node_modules/redis/index.js:223:14) at Socket.emit (node:events:513:28) at Socket.emit (node:domain:489:12) at emitErrorNT (node:internal/streams/destroy:151:8) at emitErrorCloseNT (node:internal/streams/destroy:116:3) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) original error: { "code": "CONNECTION_BROKEN",

Originally posted by @mayeulk in https://github.com/fititnt/uwazi-docker/issues/45#issuecomment-1650543697

mayeulk commented 1 year ago 
          [logs_uwazi_docker_server_stopping.txt](https://github.com/fititnt/uwazi-docker/files/12165504/logs_uwazi_docker_server_stopping.txt)

With timestamps, one sees that error 2023-07-25T20:38:37.945259424Z Error: Redis connection in broken state: connection timeout exceeded.

was just a few milliseconds before the end of the log file.

Note that, in the meantime, uwazi-docker-elasticsearch-1 and uwazi-docker-mongo-1 containers were still running

Originally posted by @mayeulk in https://github.com/fititnt/uwazi-docker/issues/45#issuecomment-1650544650

mayeulk commented 1 year ago 
          With bookworm (see https://github.com/fititnt/uwazi-docker/issues/48),  the server stopped responding, too, after a while, with the same Redis error:

[uwazi_development] uncaughtException: Redis connection in broken state: connection timeout exceeded. Error: Redis connection in broken state: connection timeout exceeded.

Originally posted by @mayeulk in https://github.com/fititnt/uwazi-docker/issues/45#issuecomment-1650673733

mayeulk commented 1 year ago

I added restart: always in the definition of the uwazi service, in docker-compose.yml. Then deleted the containers, and did docker-compose up --build --force-recreate --no-deps -d uwazi After a night, the process was still up, with my documents still there, but there was a cycle of disconnections; see last 100 rows of logs below. This was still with FROM node:18-bookworm in Dockerfile (see https://github.com/fititnt/uwazi-docker/issues/48). I may retry with FROM node:18-bullseye on another PC (in particular should you have difficulties reproducing this). Thanks.

2023-07-26T06:07:45.168333325Z     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
2023-07-26T06:07:45.168336213Z   code: 'CONNECTION_BROKEN',
2023-07-26T06:07:45.168338954Z   origin: Error: connect ECONNREFUSED 127.0.0.1:6379
2023-07-26T06:07:45.168341782Z       at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16)
2023-07-26T06:07:45.168344700Z       at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
2023-07-26T06:07:45.168347584Z     errno: -111,
2023-07-26T06:07:45.168350335Z     code: 'ECONNREFUSED',
2023-07-26T06:07:45.168353050Z     syscall: 'connect',
2023-07-26T06:07:45.168355766Z     address: '127.0.0.1',
2023-07-26T06:07:45.168358490Z     port: 6379
2023-07-26T06:07:45.168361225Z   }
2023-07-26T06:07:45.168371602Z }
2023-07-26T06:07:45.168374678Z 
2023-07-26T06:07:45.168377451Z Node.js v18.17.0
2023-07-26T06:07:45.193669376Z error Command failed with exit code 7.
2023-07-26T06:07:45.193816702Z info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2023-07-26T06:07:46.209455862Z uwazi-docker: UWAZI_GIT_RELEASE_REF: production
2023-07-26T06:07:46.209504561Z uwazi-docker: IS_FIRST_RUN: false
2023-07-26T06:07:46.209512743Z uwazi-docker: RUN_YARN_MIGRATE_REINDEX: false
2023-07-26T06:07:46.209518373Z uwazi-docker: DB_INITIALIZATION_PATH: /home/node/uwazi/database/blank_state/uwazi_development
2023-07-26T06:07:46.209523109Z uwazi-docker: Enviroment variable IS_FIRST_RUN/RUN_YARN_MIGRATE_REINDEX are not true.
2023-07-26T06:07:46.209527631Z uwazi-docker: Assume MongoDB and Elastic Search provide already are intialized.
2023-07-26T06:07:46.209532646Z uwazi-docker: [protip] is possible to initialize (or reset o initial state) MongoDB and Elastic Search with enviroment variable IS_FIRST_RUN=true
2023-07-26T06:07:46.599121891Z yarn run v1.22.19
2023-07-26T06:07:46.669828716Z $ NODE_ENV=production node server.js
2023-07-26T06:07:51.003131067Z ==> Connecting to mongodb://mongo/
2023-07-26T06:07:51.254968021Z ==> 🌎 Listening on port 3000. Open up http://localhost:3000/ in your browser.
2023-07-26T07:07:51.235314634Z 2023-07-26T07:07:51.234Z [uwazi_development] uncaughtException: Redis connection in broken state: connection timeout exceeded.
2023-07-26T07:07:51.235343958Z Error: Redis connection in broken state: connection timeout exceeded.
2023-07-26T07:07:51.235349451Z     at RedisClient.connection_gone (/home/node/uwazi/prod/node_modules/redis/index.js:588:19)
2023-07-26T07:07:51.235363023Z     at RedisClient.on_error (/home/node/uwazi/prod/node_modules/redis/index.js:346:10)
2023-07-26T07:07:51.235367147Z     at Socket.<anonymous> (/home/node/uwazi/prod/node_modules/redis/index.js:223:14)
2023-07-26T07:07:51.235370339Z     at Socket.emit (node:events:514:28)
2023-07-26T07:07:51.235372875Z     at Socket.emit (node:domain:489:12)
2023-07-26T07:07:51.235375249Z     at emitErrorNT (node:internal/streams/destroy:151:8)
2023-07-26T07:07:51.235377710Z     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
2023-07-26T07:07:51.235380800Z     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
2023-07-26T07:07:51.235383233Z [Tenant error] Error: Accessing nonexistent async context
2023-07-26T07:07:51.236672379Z 2023-07-26T07:07:51.236Z [uwazi_development] 
2023-07-26T07:07:51.236686323Z uncaught exception or unhandled rejection, Node process finished !!
2023-07-26T07:07:51.236689879Z  Error: Redis connection in broken state: connection timeout exceeded.
2023-07-26T07:07:51.236692832Z     at RedisClient.connection_gone (/home/node/uwazi/prod/node_modules/redis/index.js:588:19)
2023-07-26T07:07:51.236708500Z     at RedisClient.on_error (/home/node/uwazi/prod/node_modules/redis/index.js:346:10)
2023-07-26T07:07:51.236711742Z     at Socket.<anonymous> (/home/node/uwazi/prod/node_modules/redis/index.js:223:14)
2023-07-26T07:07:51.236714537Z     at Socket.emit (node:events:514:28)
2023-07-26T07:07:51.236716930Z     at Socket.emit (node:domain:489:12)
2023-07-26T07:07:51.236719209Z     at emitErrorNT (node:internal/streams/destroy:151:8)
2023-07-26T07:07:51.236721569Z     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
2023-07-26T07:07:51.236723914Z     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
2023-07-26T07:07:51.236726313Z original error: {
2023-07-26T07:07:51.236728622Z  "code": "CONNECTION_BROKEN",
2023-07-26T07:07:51.236731133Z  "origin": {
2023-07-26T07:07:51.236733617Z   "errno": -111,
2023-07-26T07:07:51.236736799Z   "code": "ECONNREFUSED",
2023-07-26T07:07:51.236739152Z   "syscall": "connect",
2023-07-26T07:07:51.236742072Z   "address": "127.0.0.1",
2023-07-26T07:07:51.236744419Z   "port": 6379
2023-07-26T07:07:51.236746870Z  }
2023-07-26T07:07:51.236749085Z }
2023-07-26T07:07:51.236752237Z [Tenant error] Accessing nonexistent async context
2023-07-26T07:07:51.236754876Z [Tenant error] Error: Accessing nonexistent async context
2023-07-26T07:07:51.237663361Z /home/node/uwazi/prod/app/server.js:81
2023-07-26T07:07:51.237688058Z   throw error;
2023-07-26T07:07:51.237692451Z   ^
2023-07-26T07:07:51.237696605Z 
2023-07-26T07:07:51.237699934Z Error: Redis connection in broken state: connection timeout exceeded.
2023-07-26T07:07:51.237703393Z     at RedisClient.connection_gone (/home/node/uwazi/prod/node_modules/redis/index.js:588:19)
2023-07-26T07:07:51.237706988Z     at RedisClient.on_error (/home/node/uwazi/prod/node_modules/redis/index.js:346:10)
2023-07-26T07:07:51.237711284Z     at Socket.<anonymous> (/home/node/uwazi/prod/node_modules/redis/index.js:223:14)
2023-07-26T07:07:51.237715495Z     at Socket.emit (node:events:514:28)
2023-07-26T07:07:51.237719080Z     at Socket.emit (node:domain:489:12)
2023-07-26T07:07:51.237722753Z     at emitErrorNT (node:internal/streams/destroy:151:8)
2023-07-26T07:07:51.237726617Z     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
2023-07-26T07:07:51.237739276Z     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
2023-07-26T07:07:51.237743367Z   code: 'CONNECTION_BROKEN',
2023-07-26T07:07:51.237746899Z   origin: Error: connect ECONNREFUSED 127.0.0.1:6379
2023-07-26T07:07:51.237750637Z       at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16)
2023-07-26T07:07:51.237762035Z       at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
2023-07-26T07:07:51.237766422Z     errno: -111,
2023-07-26T07:07:51.237770364Z     code: 'ECONNREFUSED',
2023-07-26T07:07:51.237772641Z     syscall: 'connect',
2023-07-26T07:07:51.237774770Z     address: '127.0.0.1',
2023-07-26T07:07:51.237776936Z     port: 6379
2023-07-26T07:07:51.237779069Z   }
2023-07-26T07:07:51.237781153Z }
2023-07-26T07:07:51.237783239Z 
2023-07-26T07:07:51.237785307Z Node.js v18.17.0
2023-07-26T07:07:51.269301619Z error Command failed with exit code 7.
2023-07-26T07:07:51.269460393Z info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2023-07-26T07:07:52.191645725Z uwazi-docker: UWAZI_GIT_RELEASE_REF: production
2023-07-26T07:07:52.191695159Z uwazi-docker: IS_FIRST_RUN: false
2023-07-26T07:07:52.191702131Z uwazi-docker: RUN_YARN_MIGRATE_REINDEX: false
2023-07-26T07:07:52.191708050Z uwazi-docker: DB_INITIALIZATION_PATH: /home/node/uwazi/database/blank_state/uwazi_development
2023-07-26T07:07:52.191935331Z uwazi-docker: Enviroment variable IS_FIRST_RUN/RUN_YARN_MIGRATE_REINDEX are not true.
2023-07-26T07:07:52.191954871Z uwazi-docker: Assume MongoDB and Elastic Search provide already are intialized.
2023-07-26T07:07:52.191961369Z uwazi-docker: [protip] is possible to initialize (or reset o initial state) MongoDB and Elastic Search with enviroment variable IS_FIRST_RUN=true
2023-07-26T07:07:52.562800540Z yarn run v1.22.19
2023-07-26T07:07:52.647406149Z $ NODE_ENV=production node server.js
2023-07-26T07:07:57.780576208Z ==> Connecting to mongodb://mongo/
2023-07-26T07:07:58.098231912Z ==> 🌎 Listening on port 3000. Open up http://localhost:3000/ in your browser.
mayeulk commented 1 year ago

Some possibly helpful hints: seems to happen every hour.

I retried with FROM node:18-bullseye. Not creating any object, just browsing the GUI now and then (twice in 3 hours). The logs got "Error: Redis connection in broken state: connection timeout exceeded." exactly one hour (to the tenth of second) after the successfull "Listening" message. Then, error every "one hour plus about 8 seconds":

2023-07-26T07:33:06.684867662Z $ NODE_ENV=production node server.js 2023-07-26T07:33:12.002041708Z ==> Connecting to mongodb://mongo/ 2023-07-26T07:33:12.396641397Z ==> 🌎 Listening on port 3000. Open up http://localhost:3000/ in your browser. 2023-07-26T08:33:12.363598113Z 2023-07-26T08:33:12.362Z [uwazi_development] uncaughtException: Redis connection in broken state: connection timeout exceeded. 2023-07-26T08:33:12.363662730Z Error: Redis connection in broken state: connection timeout exceeded. 2023-07-26T08:33:12.363676392Z

2023-07-26T08:33:12.363598113Z 2023-07-26T08:33:12.362Z [uwazi_development] uncaughtException: Redis connection in broken state: connection timeout exceeded.

Filtered logs: 2023-07-26T08:33:12.369834050Z uncaught exception or unhandled rejection, Node process finished !! 2023-07-26T09:33:20.665896284Z 2023-07-26T09:33:20.665Z [uwazi_development] uncaughtException: Redis connection in broken state: connection timeout exceeded. 2023-07-26T09:33:20.667639285Z uncaught exception or unhandled rejection, Node process finished !! 2023-07-26T10:33:26.651780039Z 2023-07-26T10:33:26.650Z [uwazi_development] uncaughtException: Redis connection in broken state: connection timeout exceeded. 2023-07-26T10:33:26.654070496Z uncaught exception or unhandled rejection, Node process finished !!

fititnt commented 12 months ago

Ok. I was suspecting that maybe either Redis or Minio would be necessary to run current versions, but since the first run I done recently it did not failed, I kept like it was.

But from your feedback, I think a good start would be add 

  redis:
    image: 'redis:5.0.14'
    container_name: uwazi-redis
    command: redis-server
    ports:
      - '6379:6379'
    environment:
      - REDIS_REPLICATION_MODE=master

from the https://github.com/huridocs/uwazi/blob/development/docker-compose.yml on this repository then wait for the error happens again