skomer
commented
5 years ago The issue appears to have been introduced in v3.1.0.
v3.0.2 works fine on the latest (master branch) version of fluentd-kubernetes-daemonset/v1.3-debian-elasticsearch-1
Open skomer opened 5 years ago
skomer
commented
5 years ago The issue appears to have been introduced in v3.1.0.
v3.0.2 works fine on the latest (master branch) version of fluentd-kubernetes-daemonset/v1.3-debian-elasticsearch-1
cosmo0920
commented
5 years ago The issue appears to have been introduced in v3.1.0.
v3.0.2 works fine on the latest (master branch) version of fluentd-kubernetes-daemonset/v1.3-debian-elasticsearch-1
I've reproduced this issue. But I got this issue for invalid ES connection configuration on configure.
export POD_NAME=$(kubectl get pods --namespace default -l "app=elasticsearch,component=client,release=my-release" -o jsonpath="{.items[0].metadata.name}")echo "Visit http://127.0.0.1:9200 to use Elasticsearch"kubectl port-forward --namespace default $POD_NAME 9200:9200docker run --rm -it fluent/fluentd-kubernetes-daemonset:v1.3-debian-elasticsearch-1 shDefault fluent/fluentd-kubernetes-daemonset:v1.3-debian-elasticsearch-1 image produces the following Fluentd configuration (w/o FLUENT_ELASTICSEARCH_XXX environment variables):
2019-03-29 11:41:38 +0900 [info]: using configuration file: <ROOT>
<source>
@type forward
</source>
<match **>
@type elasticsearch
@id out_es
@log_level "info"
include_tag_key true
host "" # ENV['FLUENT_ELASTICSEARCH_HOST']
port # ENV['FLUENT_ELASTICSEARCH_PORT']
path "" # ENV['FLUENT_ELASTICSEARCH_PATH']
scheme http
ssl_verify true
ssl_version TLSv1
user "" # ENV['FLUENT_ELASTICSEARCH_USER']
password xxxxxx # ENV['FLUENT_ELASTICSEARCH_PASSWORD']
reload_connections true
reconnect_on_error false
reload_on_failure false
logstash_prefix "logstash"
logstash_format true
index_name "logstash"
type_name "fluentd"
<buffer>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 2M
queue_limit_length 32
retry_max_interval 30
retry_forever true
</buffer>
</match>
</ROOT>Could you check your provided FLUENT_ELASTICSEARCH_XXX environment variables?
If FLUENT_ELASTICSEARCH_XXX aren't provided, ES plugin within k8s-fluentd-daemonset image dose not work correctly.
cosmo0920
commented
5 years ago Or, you should set FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR as true.
If this parameter is true, ES connection information is recreated when connection error happens.
This parameter is set as false by default.
skomer
commented
5 years ago Thank you. I no longer have access to the environment where I encountered this issue (due to moving to a new job), but I'll try to reproduce this and apply your suggestions.
dromaniuk
commented
2 years ago @skomer Any updates? :) 2,5 years have passed.
skomer
commented
2 years ago No updates.
cosmo0920
commented
2 years ago This issue seems to be quite old. I have no time to reproduce this issue. I'm closing for now. But free feel to add comments if you encounter this issue. I'll reopen when adding a new comment.
Pluggi
commented
2 years ago We had the same issue, and the cause was just that the environment variable was not set.
mokhos
commented
2 years ago I had the same issue, but turns out the issue was not because of FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR, but because the environment variables could not be passed to the elasticsearch plugin. I used the real values in the plugin and now everything is working fine.
To verfiy this, check the aggregator's logs to see if the plugin is getting the correct values. If you see empty strings or nothing in front of plugin's directives then you have the same problem as I did.
cosmo0920
commented
2 years ago Hmm…, empty environment variable is very harmful for ES plugin….
mokhos
commented
2 years ago It may be a bug on the version of fluentd that I'm using (bitnami/fluentd:1.14.5-debian-10-r1) and latest version does not have it.
Problem
Running the Docker image fluentd-kubernetes-daemonset throws an error, which appears to come from fluent-plugin-elasticsearch .
Steps to replicate
docker run --rm -it fluent/fluentd-kubernetes-daemonset:v1.3-debian-elasticsearch-1 shspits out the following error:
Other plugins installed on that image are
Expected Behavior or What you need to ask
I wondered if this were a similar issue to #443, but here the stacktrace shows that excon is v0.62.0, which is the latest.
Expected behaviour is no errors :slightly_smiling_face:
If this looks to be a problem with the Fluentd Kubernetes Daemonset image, please let me know and I'll raise it over there.
Using Fluentd and ES plugin versions