elastic / elastic-serverless-forwarder

Elastic Serverless Forwarder
Other
35 stars 36 forks source link

no input defined on cloudwatch-logs trigger #707

Closed rwmia closed 5 months ago

rwmia commented 5 months ago

I have used the SAR to create an ESF lambda application only for cloudwatch logs.

After analyzing the /aws/lambda/serverlessrepo-elastic-se log entries, there is a warning which states "no input defined".

I can't figure out why the "no input defined" is occurring. Please help :)

Below is an output log stream for an ESF invocation.

[
    {
        "@timestamp": "2024-05-02T13:23:33.799Z",
        "log.level": "info",
        "message": "trigger",
        "ecs": {
            "version": "1.6.0"
        },
        "log": {
            "logger": "root",
            "origin": {
                "file": {
                    "line": 56,
                    "name": "handler.py"
                },
                "function": "lambda_handler"
            },
            "original": "trigger"
        },
        "process": {
            "name": "MainProcess",
            "pid": 8,
            "thread": {
                "id": 139621811021632,
                "name": "MainThread"
            }
        },
        "type": "cloudwatch-logs"
    },
    {
        "@timestamp": "2024-05-02T13:23:33.799Z",
        "log.level": "info",
        "message": "config file",
        "bucket_name": "redacted",
        "ecs": {
            "version": "1.6.0"
        },
        "log": {
            "logger": "root",
            "origin": {
                "file": {
                    "line": 204,
                    "name": "utils.py"
                },
                "function": "config_yaml_from_s3"
            },
            "original": "config file"
        },
        "object_key": "config.yml",
        "process": {
            "name": "MainProcess",
            "pid": 8,
            "thread": {
                "id": 139621811021632,
                "name": "MainThread"
            }
        }
    },
    {
        "@timestamp": "2024-05-02T13:23:33.906Z",
        "log.level": "info",
        "message": "trigger",
        "ecs": {
            "version": "1.6.0"
        },
        "log": {
            "logger": "root",
            "origin": {
                "file": {
                    "line": 137,
                    "name": "handler.py"
                },
                "function": "lambda_handler"
            },
            "original": "trigger"
        },
        "process": {
            "name": "MainProcess",
            "pid": 8,
            "thread": {
                "id": 139621811021632,
                "name": "MainThread"
            }
        },
        "size": 12
    },
    {
        "@timestamp": "2024-05-02T13:23:34.346Z",
        "log.level": "warning",
        "message": "no input defined",
        "ecs": {
            "version": "1.6.0"
        },
        "input_id": "",
        "input_type": "cloudwatch-logs",
        "log": {
            "logger": "root",
            "origin": {
                "file": {
                    "line": 147,
                    "name": "handler.py"
                },
                "function": "lambda_handler"
            },
            "original": "no input defined"
        },
        "process": {
            "name": "MainProcess",
            "pid": 8,
            "thread": {
                "id": 139621811021632,
                "name": "MainThread"
            }
        }
    }
]

Below is my config.yml file in S3:

inputs:
  - type: "cloudwatch-logs"
    id: "arn:aws:logs:%AWS_REGION%:%AWS_ACCOUNT_ID%:log-group:%LOG_GROUP_NAME%:*"
    outputs:
      - type: "elasticsearch"
        args:
          elasticsearch_url: "redacted"
          username: "redacted"
          password: "redacted"
          es_datastream_name: "logs-generic-default"
          batch_max_actions: 500
          batch_max_bytes: 10485760
  - type: "cloudwatch-logs"
    id: "arn:aws:logs:%AWS_REGION%:%AWS_ACCOUNT_ID%:log-group:%LOG_GROUP_NAME%:log-stream:%LOG_STREAM_NAME%"
    outputs:
      - type: "elasticsearch"
        args:
          elasticsearch_url: "redacted"
          username: "redacted"
          password: "redacted"
          es_datastream_name: "logs-generic-default"
          batch_max_actions: 500
          batch_max_bytes: 10485760
rwmia commented 5 months ago

Disregard, it seems you must provide the arn for each log group, instead of %XX% in the config.yaml