search

dbhowell / pino-cloudwatch

AWS CloudWatch Logs transport for pino
MIT License
38 stars 13 forks source link

pino cloudwatch stop sending log when ulimit error #36

Open samuraitruong opened 3 years ago

samuraitruong commented 3 years ago

Recently we have a bug in production that causes too many open file (ie ulimit). When this happens our application still works with some random error. however, the logs will stop sending to cloudwatch until we restart the server

Is there any config that lets pino keep re-try if there any network/tcp error.

dbhowell commented 3 years ago

Good question, and I'm not sure how it would resolve on a ulimit.

I assume the error log didn't make it to CloudWatch?

samuraitruong commented 3 years ago

@dbhowell

in general, when we have any error with tcp connection, then pino will stop working,

We are now fix the limit issue but when we run load testing and the application creating huge amount of logs then pino stop sending logs to cloudwatch even after we stop testing and the application back to normal traffic.

Is there anyway to configure re-try or keep alive incase of pino get an error?

demian85 commented 3 years ago

I'm facing the same issue. I need to restart the server to resume cloudwatch logging.