Open mafazely opened 8 months ago
Any recommendations?
Same here. id_key
and remove_keys
are ignored by elasticsearch_data_stream
.
ElasticSearch 8.5
Fluentd 1.16.4 (container)
I would also need this functionality. Is it true that this feature is not supported for data streams or are we missing some configuration?
And is there a technical reason for this or has just no one had the time to implement it so far?
Problem
I would like to use the gen_id feature to avoid duplicate documents in Elasticsearch. Currently, I employ Fluent Bit on my servers to read and parse logs. These logs are then sent to Fluentd, acting as an aggregator, which subsequently pushes them to the Elasticsearch cluster. However, I've encountered an issue where the id_key doesn't function as expected on the elasticsearch_data_stream. This results in the addition of a _hash field to the documents in Elasticsearch, but it is not being recognized as the _id field.
Steps to replicate
Expected Behavior or What you need to ask
Use _hash as id field for checking the uniqueness of docs in elasticsearch but didn't do this.
Using Fluentd and ES plugin versions