pnxtech / hydra

A light-weight library for building distributed applications such as microservices
https://www.hydramicroservice.com
MIT License
645 stars 54 forks source link

Nodejs crashes when redis connection dies - How to handle it ? #199

Open FDT2k opened 6 years ago

FDT2k commented 6 years ago

Hi,

I'm considering using hydra for a small project but I have a big problem. When I restart redis, all the related microservices crashes, without retrying the connection

I tried to tweak the retry_strategy but I'm a bit lost.

How can I catch theses exceptions or prevent them ?

log entry { ts: '2018-06-13T09:36:04.251Z', serviceName: 'my-awesome-microservice', type: 'error', processID: 1953, msg: 'Established Redis server connection has closed' } Unhandled Exception { Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 6379 } Unhandled Exception { Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14) errno: 'ECONNREFUSED', code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 6379 } log entry { ts: '2018-06-13T09:36:04.458Z', serviceName: 'my-awesome-microservice', type: 'error', processID: 1953, msg: 'Reconnecting to Redis server...' } Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)

Thank you in advance.

cjus commented 6 years ago

@FDT2k Yes and that's by design. In a containerization environment (we use Docker Swarm) a Restart of Redis would cause services to restart. There is a retry period in Hydra which is supposed to retry connecting to Redis for a period. We are aware that this restart logic needs some work. PRs and feedback are definitely welcomed!