[httpendpointexporter] Can we provide a new exporter for exporting logs to a http endpoint?

[Dongqi-Guo]

Is your feature request related to a problem? Please describe.

Hello guys, since I worked on app logs collect, I deployed fluentbit with my application to collect logs, then I use fluentbit's output to write logs to otel collector, I export the logs to loki, also I wanna export to my other applications which expose http service(like 'http://example:8080/collect/logs'), but I can not find any exporter supported this. any idea about this? appreciate it!

Describe the solution you'd like

I think there are at least two ways to solve this:

1. a new export supports export to a http endpoint
2. any other exporter already support this feature?

Describe alternatives you've considered

I found a http_forward extension that supports forward fluentbit http output, but I think an exporter should be a better solution

Additional context

No response

[evan-bradley]

What format(s) are you looking to export logs in? Exporters handle sending telemetry data in the format used by the backend, so you will need an exporter specific to the format you are looking to use. To export to multiple backends, you can simply list multiple exporters in your telemetry pipeline configurations, like this.

[Dongqi-Guo]

@evan-bradley thx for reply, my spring web app accepts jsonarray format req to compress and dispatch to another destination. I wonder if there is a export could achieve this?

[evan-bradley]

Are you using a custom JSON schema, or is it a standard schema type? There are no exporters that currently support writing log information to a custom schema type, though it would be possible to implement. If you are using a standard format, it would depend on whether any exporters support that format.

If your app is flexible with regards to what format you receive logs in, the OTLP HTTP exporter would probably be the best solution, though currently it sends OTLP over Protobuf instead of JSON.

[Dongqi-Guo]

@evan-bradley thx, got it. one more thing, is there any plan to implement this feature which already had in fluentd? In such case, otel collector just like a "forwarder" between fluentbit and spring application accepts http request.

[evan-bradley]

The HTTP forwarder extension is probably the closest thing that exists currently, though it is unmaintained.

If you wanted to export data in the Fluent forward format in a pipeline, there would need to be a Fluent forward exporter to achieve that functionality. The reason that requests can't be simply proxied through in a pipeline is because they are deserialized from the format sent to the Collector into the Collector's pipeline data format, then are serialized when the Collector exports the telemetry data to the desired backend. So in this case the data would have to be serialized into the Fluent forward format when exporting to the application. If this is functionality you would be interested in contributing, you can open a request to add a new component and see if someone would be willing to sponsor it.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[atoulme]

You can export with splunk_hec with the option to export as raw. That should do the job.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[atoulme]

Closing this issue as we have both responded - please let us know if those responses help you, or if you need to reopen the issue. Thanks!