search

kube-logging / logging-operator

Logging operator for Kubernetes
https://kube-logging.dev
Apache License 2.0
1.52k stars 326 forks source link

Google Stackdriver output #198

Open seanson opened 4 years ago

seanson commented 4 years ago

Is your feature request related to a problem? Please describe. We use stackdriver as an output for fluentd for a manual deploy currently but would like to move to logging-operator.

Describe the solution you'd like I'd like to use the fluent-plugin-google-cloud in the same manner as the other backend based plugins like ElasticSearch.

Describe alternatives you've considered I have considered a GCS logging bucket and a function to ship the logs from there into the StackDriver API but it is a less than ideal option.

Additional context This was previously raised in #40 but considered not required unless someone was willing to raise it as a separate request later.

KrustyHack commented 4 years ago

Ay,

Is StackDriver output available ? We want to move from our old logging workflow to Banzai Cloud Logging Operator but StackDriver is mandatory for us.

aslafy-z commented 2 years ago

This becomes a requirement for us too. Can we think of rescheduling this task?

ahma commented 2 years ago

@aslafy-z okay, I'll check it.

ekhaydarov commented 1 year ago

Any updates? Would love if this became available

ekhaydarov commented 1 year ago

@tarokkk would you be able to provide an update at all please? Just so we can assess to wait or use a different tool. Thanks

tarokkk commented 1 year ago

This issue is undeservedly staled We are revamping the fluentd image building so we are taking a look at this now!

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

pepov commented 1 year ago

The problem, why this has not yet been implemented is the complexity it brings into the already complex and big fluentd image.

One idea I have at the moment is to create a base image that includes the base components only without any third party outputs, so that it is small and faster to build. Users of the the operator can than depend on this base image and add their combination of output plugins on top of that, which would work for their specific use case.

The only thing that bothers me in this case is that we cannot easily decide from the logging operator if an output is supported by the image or not, so if someone sees the google stackdriver output configuration could naively assume, that it is supported in the base image and they would be surprised, that they have to build their own.

Any thoughts?

stale[bot] commented 10 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

ekhaydarov commented 10 months ago

Hi @pepov, THanks for taking a look. II'm pretty sure its not possible but a bare base image with no outputs would be quickets, then at runtime install additiona outputs that the end user needs.?

pepov commented 10 months ago

@ekhaydarov we don't have a method to install plugins at runtime, if someone needs more outputs they have to build a custom image or they will be able to run multiple aggregators (through multiple logging resources) and use different images on each.

pepov commented 9 months ago

There is another option which some of the core maintainers would be open to explore, which is implementing an output for the syslogng based aggregator.

There are two possible solutions in that case as well: migrate to the syslog-ng based aggregator or use a separate syslog-ng logging without a fluentbit agent and relay logs designated in stackdriver into that aggregator.

Let me know if any of the above would work for you.

pepov commented 2 months ago

With the introduction of the new multi-stage image build it is now possible to have an image that supports Google Stackdriver only without having to pull the ruby dependency hell into our full image, which is already quite large. Given this, we can finally implement the required API in the Logging Operator as well. In case someone would need an image that contains not just the Stackdriver output they will have the option to base off of our stackdriver image quite easily.

pepov commented 1 month ago

anyone still interested in this?