kiwigrid / helm-charts

Helm charts for Kubernetes curated by Kiwigrid
https://kiwigrid.github.io
MIT License
184 stars 210 forks source link

Specify primary shard and replica count in es fluentd helm chart #390

Closed Sushant20 closed 4 years ago

Sushant20 commented 4 years ago

Is this a request for help?: Yes


Is this a BUG REPORT or FEATURE REQUEST? (choose one): Question/Feature request

Version of Helm and Kubernetes: helm2 , eks 1.15

Which chart in which version: fluentd-elasticsearch : 4.9.3

What happened: Elasticsearch creates five primary shards and one replica for each index

What you expected to happen: Looking for option to set the primary shard count to 1 in the helm chart or configmap

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

monotek commented 4 years ago

This would be a setting in elastic search, not fluentd.

Sushant20 commented 4 years ago

@monotek I am using AWS Elasticsearch and ideally primary shard and replica count setting is done during index creation because you might want to set this differently for different indexes. I am looking for something similar to what logstash output plugin provides: https://discuss.elastic.co/t/how-do-you-change-the-default-number-of-shards/1508/4. I believe, fluentd plugin is creating index so there should be setting at index level from fluentd end and not elasticsearch. Request you to not close the issue unless and until we get reply from issue creator or there is no reply from last 1 day

monotek commented 4 years ago

So you wan't to have other settings as your es default?

Sushant20 commented 4 years ago

Yes @monotek , I need to set other setting as index default, similar to how logstash plugin provides: http://spuder.github.io/2015/elasticsearch-default-shards/

monotek commented 4 years ago

not sure if fluentd supports this. if so, should be rather easy to implement.

Sushant20 commented 4 years ago

@monotek Could you please recheck at your end, I don't see an option to set it. This is definitely good feature, if not already present. Please reopen the issue, if its not available

monotek commented 4 years ago

Please check by yourself. I'm on vacation till middle of September.