search

fluent / fluent-bit-go

Fluent Bit Golang package to build plugins
Apache License 2.0
189 stars 52 forks source link

Is there a way to get the chunk ID at the custom GO output plugin level ? #67

Open Nicusor97 opened 1 year ago

Nicusor97 commented 1 year ago

I'm looking for a way to get the chunk ID (which I suppose is unique per chunk) at a custom GO output plugin level. I'm using Fluent-Bit with an Tail input plugin (default one) and a GO custom output plugin that is used to forward the data to specific output.

Our infrastructure is based on retries and we adding some audits logs where we log each line from a chunk just to be collected by a log agent and pushed to Kibana. Since the same chunk is retried multiple times, I want to somehow get the unique ID of that specific chunk, store it in memory and at the next validations make sure the same ID is used in logs so we can filter by that specific space ID.

Long story short, these are my questions:

  1. Is there a way to get the chunk ID at a custom GO output plugin level considering I'm using an TAIL input plugin ?
  2. Is there is no way to get the chunk ID, do you have any suggestion for how to map the chunk to something unique so I will be aware of the chunk's unicity when I log the action (retry, fail, sent)?

Thank you!

l2dy commented 1 year ago

For output to Elasticsearch, you could use the native plugin which supports Generate_ID or Id_Key to prevents duplicate records.

Nicusor97 commented 1 year ago

Ah, I think I was not clear enough. My custom Output plugin is perform HTTP calls to a Web Service with batches of events. Each time when a action is performed I do print a log into a file that is scraped by another agent which is forwarding the logs generated by the Output plugin in OpenSearch. At the output plugin level there is nothing related to OpenSearch.

l2dy commented 1 year ago

The simple answer is you can't. Only a tag and data, which includes timestamp and optional metadata, can be passed to a Go plugin. https://github.com/fluent/fluent-bit/blob/master/src/proxy/go/go.c#L138-L143