hrxi / tracing-loki

A tracing layer for shipping logs to Grafana Loki
Apache License 2.0
43 stars 20 forks source link

Question: Possible to not miss logs without sleep? #9

Closed chrkau closed 1 year ago

chrkau commented 2 years ago

I'm pretty new to Rust so sorry if this should be obvious.

Looking at the example given in examples it calls tokio::time::sleep(Duration::from_secs(1)).await;. Now if I remove that line my loki instance will not receive a log. In my understanding that is because the background reporting thread is killed when the program exits - before it has a chance to deliver the log to loki.

So my question is: Is there a way to ensure all logs have been delivered before exiting the program? Thanks.

hrxi commented 2 years ago

I currently don't have a good solution for that.

I'd guess one could provide a function to wait for any outstanding log request. Would that suit your needs?

chrkau commented 2 years ago

Sure that would be great.

I've also played around with sentry-tracing in the mean time and from what I gather is they use the guard pattern to flush the reporting queue on exit. I'm in no position to say if that is a good/better idea but I thought I'd mention it in case you hadn't considered it yet.

Veetaha commented 1 year ago

I think the easiest solution is to provide an abort token like that:

fn layer_with_abort(/* params as for regular layer() */, abort: impl Future + Send) -> (Layer, BackgroundTask);

Then the BackgroundTask could store the abort future (potentially box it to erase the type), and use that in in its processing loop to stop processing and flush the buffers, send data to loki when that abort future is Ready.

Note that we may want to specify the timeout value. If we aren't able to flush the buffers and send data to loki for the given timeout duration, then we give up and cancel that activity. The timeout could be specified as a yet another parameter of layer_with_abort. It may make sense to accept a config struct / make a builder instead of positional params to do that then

CryoMyst commented 1 year ago

Isn't this just a WorkerGuard? https://docs.rs/tracing-appender/latest/tracing_appender/non_blocking/struct.WorkerGuard.html

hrxi commented 1 year ago

Fixed by #18.