search

atomita / fluent-plugin-aws-elasticsearch-service

This repository is extends of https://github.com/uken/fluent-plugin-elasticsearch/ which made connectable to Amazon Elassticsearch Service using Aws Signers V4. Please check the requirements to connect to Amazon Elassticsearch Service without using Aws Signers V4 for before starting use. (reference https://groups.google.com/forum/#!msg/fluentd/uW87VAOqxeE/cfhenicEBAAJ)
MIT License
122 stars 49 forks source link

Error Starting Fluentd #74

Open wooxudong opened 2 years ago

wooxudong commented 2 years ago

Problem

The fluend container would not be able to start with following errors

Screenshot 2021-09-22 at 5 43 01 PM

...

Steps to replicate

FROM bitnami/fluentd:1.14.0

RUN fluent-gem install 'fluent-plugin-aws-elasticsearch-service' -v 2.4.1

Expected Behavior or What you need to ask

Fluentd to start correctly ...

Using Fluentd and ES plugin versions

Fluentd: v1.14.0 fluent-plugin-aws-elasticsearch-service: v 2.4.1

pavel-trubitsyn commented 2 years ago

Hi, I have the same issue with fluent/fluentd:v1.14.1.

pavel-trubitsyn commented 2 years ago

Solution https://github.com/uken/fluent-plugin-elasticsearch/issues/912#issuecomment-894242260

wooxudong commented 2 years ago

@pavel-trubitsyn , thanks for the solution.

I think it shall be a proper fix for this plugin to solve the dependency chain ?

wooxudong commented 2 years ago

Any updates on this ?

DaemonDude23 commented 2 years ago

I've been watching this problem overall for the last year. Today I got it working after learning about an OpenSearch-specific plugin from the Fluentd maintainers.

I had previously built a container with plugins from this repo to push logs to AWS's OpenSearch, but when trying to update it, ran into these dependency conflicts. I upgraded from: fluent/fluentd:v1.13.3-debian-1.0 to fluent/fluentd:v1.14.5-debian-1.0. The Dockerfile I kept the same, other than removing any elasticsearch plugins and replacing it with: fluent-plugin-opensearch:1.0.1 (https://github.com/fluent/fluent-plugin-opensearch).

Then I modified the config to change the output @type aws-elasticsearch-service to opensearch. I removed 2 parameters that are not supported by this plugin (log_es_400_reason and suppress_type_name) and it's working just fine. I see there are many issues in this repo that relate to this problem. Hopefully this will help someone resolve it. EDIT: There's a bug with the Opensearch plugin: https://github.com/fluent/fluent-plugin-opensearch/issues/24