datalust / pino-seq

A stream to send Pino events to Seq
Apache License 2.0
11 stars 10 forks source link

Implement flushSync to flush logs when running inside of Cloud functions #33

Open egorpavlikhin opened 7 months ago

egorpavlikhin commented 7 months ago

When flushing logs pino doesn't actually call _final and wait for the logs to flush when using streams. Instead it calls a poorly documented flushSync method that is not part of the stream.Writable interface. As the name suggests the function is not async and will block the event loop until the logs are flushed. Normally this wouldn't be an issue, but when running inside a Cloud function the function will often exit before the logs are flushed, so this implementation is in line with what pino recommends https://getpino.io/#/docs/asynchronous?id=aws-lambda

https://github.com/pinojs/pino/blob/c109804c6ce4b4545f3dfe1d021ceb698582fe15/lib/multistream.js#L82

In practice, without this change we had about 20% of our logs disappear as pino-seq would still be trying to send logs to seq when our Azure functions has already finished executing (eventually failing after a 30 seconds timeout).

How to review: I've been trying to find a good way to run async functions synchronously in javascript but couldn't. There are libraries that spawn a separate nodejs process and wait for completion there, but since this library doesn't necessarily run in nodejs context I thought it's a bit of an overkill. Happy to hear thoughts on how this should ideally look like.

liammclennan commented 7 months ago

Hi @egorpavlikhin,

It looks like you should be able to just await _logger.flush() or _logger.close()

richard-stafflink commented 7 months ago

Hi! I would really love to get a hold of this

richard-stafflink commented 7 months ago

Hi @egorpavlikhin,

It looks like you should be able to just await _logger.flush() or _logger.close()

Could you expose the _logger on the .d.ts please?