Closed yoav-klein closed 1 year ago
I'm getting the same issue
Ok I just found a fix;
You need to create a ConfigMap to override the fluent.conf file and mount it to the DaemonSet
Config contents taken from https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.16/debian-opensearch/conf/fluent.conf, I've just removed the following lines, as we don't need to mount these files when using cloud OpenSearch in AWS
ca_file "#{ENV['FLUENT_OPENSEARCH_CA_PATH']}"
client_cert "#{ENV['FLUENT_OPENSEARCH_CLIENT_CERT']}"
client_key "#{ENV['FLUENT_OPENSEARCH_CLIENT_KEY']}"
client_key_pass "#{ENV['FLUENT_OPENSEARCH_CLIENT_KEY_PASS']}"
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd
namespace: logging
data:
fluent.conf: |
@include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf"
@include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || 'prometheus'}.conf"
@include kubernetes.conf
@include conf.d/*.conf
<match **>
@type opensearch
@id out_os
@log_level info
include_tag_key true
host "#{ENV['FLUENT_OPENSEARCH_HOST']}"
port "#{ENV['FLUENT_OPENSEARCH_PORT']}"
path "#{ENV['FLUENT_OPENSEARCH_PATH']}"
scheme "#{ENV['FLUENT_OPENSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_OPENSEARCH_SSL_VERIFY'] || 'true'}"
ssl_version "#{ENV['FLUENT_OPENSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
user "#{ENV['FLUENT_OPENSEARCH_USER']}"
password "#{ENV['FLUENT_OPENSEARCH_PASSWORD']}"
index_name "#{ENV['FLUENT_OPENSEARCH_INDEX_NAME'] || 'fluentd'}"
logstash_dateformat "#{ENV['FLUENT_OPENSEARCH_LOGSTASH_DATEFORMAT'] || '%Y.%m.%d'}"
logstash_format "#{ENV['FLUENT_OPENSEARCH_LOGSTASH_FORMAT'] || 'false'}"
logstash_prefix "#{ENV['FLUENT_OPENSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
logstash_prefix_separator "#{ENV['FLUENT_OPENSEARCH_LOGSTASH_PREFIX_SEPARATOR'] || '-'}"
<buffer>
flush_thread_count "#{ENV['FLUENT_OPENSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '1'}"
flush_mode "#{ENV['FLUENT_OPENSEARCH_BUFFER_FLUSH_MODE'] || 'interval'}"
flush_interval "#{ENV['FLUENT_OPENSEARCH_BUFFER_FLUSH_INTERVAL'] || '60s'}"
chunk_limit_size "#{ENV['FLUENT_OPENSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '8M'}"
total_limit_size "#{ENV['FLUENT_OPENSEARCH_BUFFER_TOTAL_LIMIT_SIZE'] || '512M'}"
retry_max_interval "#{ENV['FLUENT_OPENSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
retry_timeout "#{ENV['FLUENT_OPENSEARCH_BUFFER_RETRY_TIMEOUT'] || '72h'}"
retry_forever "#{ENV['FLUENT_OPENSEARCH_BUFFER_RETRY_FOREVER'] || 'false'}"
</buffer>
</match>
You then need to mount the ConfigMap like this
apiVersion: apps/v1
kind: DaemonSet
spec:
template:
spec:
containers:
- name: fluentd-logging
# ...
volumeMounts:
# ...
- name: fluent-config
mountPath: /fluentd/etc/fluent.conf
subPath: fluent.conf
volumes:
# ...
- name: fluent-config
configMap:
name: fluentd
@Jared0430 , thanks ! found the same fix also :)
Seems to be solved. Closing.
(check apply)
Steps to replicate
Expected Behavior or What you need to ask
I expect that fluentd will send logs to OpenSearch, but instead I get the following error:
...
Using Fluentd and OpenSearch plugin versions
fluent-plugin-opensearch 1.0.10 fluentd 1.15.3
fluentd --version
ortd-agent --version
fluent-gem list
,td-agent-gem list
or your Gemfile.lock