search

atomita / fluent-plugin-aws-elasticsearch-service

This repository is extends of https://github.com/uken/fluent-plugin-elasticsearch/ which made connectable to Amazon Elassticsearch Service using Aws Signers V4. Please check the requirements to connect to Amazon Elassticsearch Service without using Aws Signers V4 for before starting use. (reference https://groups.google.com/forum/#!msg/fluentd/uW87VAOqxeE/cfhenicEBAAJ)
MIT License
122 stars 49 forks source link

Please add log_400_event or log_level (debug) options #81

Open AlKapkone opened 2 years ago

AlKapkone commented 2 years ago

Problem

2022-04-01 11:51:33 +0000 [warn]: #0 send an error event to @ERROR: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="app.fpm" time=2022-04-01 11:51:26.872353958 +0000

I want to know the reason why, so I need the log_level debug or at least the log_es_400_reason parameter. Like we have it here - https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/elasticsearch/#elasticsearch-log_es_400_reason

Using Fluentd and ES plugin versions

    repository: ghcr.io/banzaicloud/fluentd
    tag: v1.14.4-alpine-2
AlKapkone commented 2 years ago

@cosmo0920 Hello, sorry to bother you, but do we have a way how to troubleshoot my issue? I just want to know why AWS OpenSearch reject my logs. Thanks.

ServerNinja commented 2 years ago

We are running into an issue where this option would be very helpful.

AlKapkone commented 2 years ago

Hi, @ServerNinja. It looks like this repo is "dead". Unfortunately, our request will not be heard.

cosmo0920 commented 2 years ago

Already inherited: https://github.com/atomita/fluent-plugin-aws-elasticsearch-service/blob/master/lib/fluent/plugin/out_aws-elasticsearch-service.rb#L10

AlKapkone commented 2 years ago

Already inherited: https://github.com/atomita/fluent-plugin-aws-elasticsearch-service/blob/master/lib/fluent/plugin/out_aws-elasticsearch-service.rb#L10

Hi, @cosmo0920 . Actually, I still didn't know how to enable debug and use this log_es_400_reason parameter. Could you help me with that? My config is next now:

      awsElasticsearch:
        endpoint:
          url: https://logs.example.com
          region: us-east-1
          assume_role_arn:
            value: arn:aws:iam::12345678:role/logs-full-access
        index_name: app-${kube_app_name}-%Y.%m.%d
        buffer:
          tags: tag, time, kube_app_name
          timekey: 3s
          timekey_wait: 5s
          timekey_use_utc: true
          flush_thread_count: 8
AlKapkone commented 2 years ago

I'll try to clarify and add more context: I use banzai-cloud logging-operator - https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/aws_elasticsearch/ I use the fluent-plugin-aws-elasticsearch-service plugin as an output for my logs.

AlKapkone commented 2 years ago

So I suppose this log_es_400_reason option should be added from the banzai logging-operator side? Am I right ?

cosmo0920 commented 2 years ago

Yes. Adding log_es_400_reason parameter support is required here: https://github.com/banzaicloud/logging-operator/blob/c9722ee319584ff75ebda49ef4095b6bf127d7d4/pkg/sdk/model/output/aws_elasticsearch.go Like as this PR: https://github.com/banzaicloud/logging-operator/pull/858

cosmo0920 commented 2 years ago

And this issue is not related this repo but, for tracking issue, we leave this issue as-is for now.