in_elasticsearch with AWS for Fluent Bit, plugin name that don't exists

ibrahimjelliti commented 11 months ago

Describe the question/issue

I am running AWS for Fluent Bit on EKS to forward logs to cloudwatch, I am trying to ship logs of Elasticsearch running on eks to cloudwatch. but I am getting an error

Configuration

  open-search.conf: |
    [INPUT]
        Name             elasticsearch
        Tag                 opensearch.firstlog
        listen              0.0.0.0
        port                9200
        buffer_max_size     20M
        buffer_chunk_size   10M

    [OUTPUT]
        Name                cloudwatch_logs
        Match               opensearch.*
        region              us-east-1
        log_group_name      /aws/containerinsights/eks/opensearch
        log_stream_prefix   opensearch-
        auto_create_group   true
        extra_user_agent    container-insights

Fluent Bit Log Output

Fluent Bit v1.9.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/21 16:41:16] [error] [config] section 'elasticsearch' tried to instance a plugin name that don't exists
[2023/07/21 16:41:16] [error] configuration file contains errors, aborting.
AWS for Fluent Bit Container Image Version 2.31.11

Fluent Bit Version Info

public.ecr.aws/aws-observability/aws-for-fluent-bit:stable -->

Cluster Details

Application Details

I am running Opensearch on EKS and having logs written in the OS log

Steps to reproduce the issue

use the Configuration section above in Fluentbit on eks,

PettitWesley commented 11 months ago

[INPUT]
    Name             elasticsearch

Elasticsearch is not an input, its an output:

[OUTPUT]
     Name elasticsearch

https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch

ibrahimjelliti commented 11 months ago

Hi @PettitWesley, I am referring to this page https://docs.fluentbit.io/manual/pipeline/inputs/elasticsearch. nevertheless, I was looking at the C code of the in_elasticsearch, I think this is related to the logging of the _Bulk API while I am looking to forward opensearch logs found at /usr/share/opensearch/logs inside the pod. is there a way to have fluentbit tail a json file that is inside a pod?

PettitWesley commented 11 months ago

@ibrahimjelliti Sorry, I didn't realize there was an input for elasticsearch too. It looks like that was added in 2.x, which is not supported in this distro yet. Please post a note here to request 2.x support in AWS distro: https://github.com/aws/aws-for-fluent-bit/issues/494

forward opensearch logs found at /usr/share/opensearch/logs inside the pod. is there a way to have fluentbit tail a json file that is inside a pod?

Yes! Configure a tail input to read log files: https://docs.fluentbit.io/manual/pipeline/inputs/tail

And make sure that Fluent Bit has access to the location on the filesystem. If the logs are created in another container, make sure to set up a volume mount to Fluent Bit.

aws / aws-for-fluent-bit