Ship Logs to Multiple Destinations

defenseunicorns / uds-core

A secure runtime platform for mission-critical capabilities

https://uds.defenseunicorns.com

Apache License 2.0

43 stars 18 forks source link

Ship Logs to Multiple Destinations #377

Open ntwkninja opened 4 months ago

ntwkninja commented 4 months ago

Is your feature request related to a problem? Please describe.

As a platform admin of UDS-Core, I need to ship logs to multiple locations

Describe the solution you'd like

Given a uds-core deployment
When configuring logging
Then I have the ability to ship logs to select multiple destinations (i.e. grafana + SIEM, grafana + S3, etc.)

Describe alternatives you've considered

adding & integrating an additional logging capability

Additional context

This is a requirement for most environments we support

mjnagel commented 4 months ago

cc @MxNxPx - I know we discussed this need as well. It seemed like two promising options were:

Fluentbit (instead of promtail): It supports multiple outputs, of a number of flavors. We have heard from users that fluentbit can be challenging to work with and debug in some environments though (buffer issues?).
OTel (new addition): With OTel we could "spoof" a Loki instance and have promtail ship to Loki + OTel. Then from OTel we could use any of a number of exporters to ship the logs to an additional output (including some SIEM tools and S3).

OTel seems more promising at first glance and may have some other benefits as we look at adding Tempo for tracing? Will definitely require some evaluation/testing. It also might be a good case for a default, but optional, package? Or maybe is a common enough requirement that it should be enabled always.

mjnagel commented 3 months ago

Grafana Alloy (and OTel collector) now supports S3 exports as of https://www.github.com/grafana/alloy/pull/730 being merged 🙌 . If we want to stay with the Grafana ecosystem this might be a good option, and seems like we might be able to drop promtail if we use alloy 👀 - cc @jimmy-ungerman

adam-defenseunicorns commented 2 months ago

@mjnagel will you do the scoping for an experiment here?

mjnagel commented 2 months ago

Scoping the initial work here as a spike/experiment around otel...

build a component/package for otel "community" or grafana alloy
validate config to ship logs to loki with this
test if removing promtail is feasible and/or desirable (does OTel handle the scraping for us)
evaluate export options (s3 seems like an easy win, but are there other good export options for us)
initial 👍/👎 on whether this seems like a good path forward or if we should look elsewhere to solve this problem

Out of scope here:

using other tools beyond otel/alloy
removing loki (initially we want to focus on augmenting loki functionality, not replacing it)

mjnagel commented 1 month ago

Draft PR opened swapping Promtail -> Alloy. It seems viable but going to seek some feedback from other devs + users around whether this is a desirable swap or if we should evaluate other tools like vector further.

mjnagel commented 3 weeks ago

@ntwkninja could you clarify on this ask - are their specific targets you would like to ship logs to? We have done work to get Alloy ready in UDS Core with direct/easy support for AWS S3 exporting. Were there other export locations you had in mind that native integrations with would be beneficial for? It looks like the Splunk Exporter for example is something that the Grafana team may be willing to include/support based on this issue but it is not currently implemented.

cc @brianrexrode

mjnagel commented 2 weeks ago

Spending some additional cycles here to evaluate vector further. It has a number of appealing aspects like direct integration with all currently desired "sink" locations (Splunk, Elastic, S3) and seems to have a flexible and simple configuration.