search

Nebo15 / logger_json

JSON logger formatter with support for Google Cloud, DataDog and other for Elixir.
https://nebo15.github.io/logger_json/
MIT License
237 stars 92 forks source link

Usage of "device" undocumented #68

Closed Catharz closed 3 years ago

Catharz commented 3 years ago

I'm already using LoggerJSON for our cloudwatch logs, but have written my own logger for Datadog.

However, I'm wondering about the "device" config option. I think it would be easier for me to create a "device" to send logs to Datadog.

Is it safe to use this undocumented feature?

AndrewDryga commented 3 years ago

Hello @Catharz, did you mean that you want to send logs directly to DataDog using their API? The :device option is described here (logger_json.ex code is almost a copy-paste from Elixir logger console backend). So :device it's basically an Elixir/Erlang IO-device and I'm not sure it's a good fit to put an API client there, even if you manage to create a sort of virtual device API calls are slow and it can be challenging to do proper batching for such logs, at least this library won't help you with that at all.

We never needed such functionality because our services are running on k8s so we just write everything to stdout and then with fluentd we can ingest them anywhere we want (datadog, stackdriver and pretty much anything else). Is there any reason why you want to ingest them directly from Elixir?

From the top of my head I would suggest a couple of options:

  1. You can copy logger_json.ex and instead of using an IO device do some API calls there. Make sure you don't make a call per log entry because your logging would slow down the system and part of logs would be auto-discarded;
  2. Take look at new :erlang logger, they have backends and maybe all you want is just a backend for datadog.
  3. Write JSON to a file and then use other tool to send logs to the third-party service.
  4. We can refactor LoggerJSON but that can be a long path to take :).
Catharz commented 3 years ago

Thanks for the detailed response @AndrewDryga. This is the first time I've delved into the depths of logging in Elixir or Erlang, so still getting up to speed.

Yes, we're sending via the Datadog API, but only "error" events. And sorry, it was a backend I wrote, not a logger.

We're also running everything on k8s, but everything on standard out goes to cloudwatch. Our log volumes are way too high to send everything to Datadog. What I've spiked so far is my own backend at error level, sending error reports there as an alternative to Rollbar. That way we can see stack traces and error logs in the context of our trace data.

AndrewDryga commented 3 years ago

Oh we have some instances on AWS too, the way how you could solve it is modify the standard DataDog lambda function and make sure it only sends errors (from all cloudwatch logs). We send millions of logs from CloudWatch to DataDog using such lamba.

Catharz commented 3 years ago

Thanks for your help @AndrewDryga, I consider this closed now.