search

logdna / logger-node

A nodejs logger client for LogDNA
MIT License
34 stars 17 forks source link

LogDNA outage seems to cause high latency in monitored servers #50

Closed Sytten closed 2 years ago

Sytten commented 3 years ago

Since the outage started on April 28, we are seeing latency spikes on our servers. Might be unrelated, but when we disabled the winston log transport it seems to have solved our issue. This leads to me to think that something is probably blocking the event loop if the sending is slowed down.

darinspivey commented 3 years ago

That's an interesting observation. We will definitely investigate this oddity, although it might be hard to reproduce if it's related to degraded performance. Nevertheless, we'll check it out.

Sytten commented 3 years ago

Since that comment we received confirmation that GCP also had an outage that caused high latency in network requests. So it could be unrelated but I wonder if there is a possibility that the logger created pression on the network and/or made some calls sync.

darinspivey commented 3 years ago

There are definitely no synchronous calls. It should spool a ton of entries/requests before dying from a buffer overflow (but the buffer is very large--5MB). We're looking into our architecture and will be able to prove or disprove this at some point. We're on it 👍🏻

Sytten commented 3 years ago

Thanks for the confirmation :)

darinspivey commented 2 years ago

Closing this due to inactivity and lack of a tangible problem to solve.