elastic / elastic-serverless-forwarder

Elastic Serverless Forwarder
Other
35 stars 36 forks source link

Deployment method using AWS Console doesn't work #732

Closed tetianakravchenko closed 1 month ago

tetianakravchenko commented 3 months ago

I followed documentation - https://www.elastic.co/guide/en/esf/current/aws-deploy-elastic-serverless-forwarder.html#aws-serverless-forwarder-deploy-console

during installation - I've set only ElasticServerlessForwarderS3ConfigFile in format s3://<config-bucket-name>/config.yaml

inputs:
  - type: "cloudwatch-logs"
    id: "arn:aws:logs:...:...:log-group:/aws/lambda/name-test:*"

    outputs:
      - type: "elasticsearch"
        args:
          # either elasticsearch_url or cloud_id, elasticsearch_url takes precedence if both are included
          elasticsearch_url: "https://..."
          # cloud_id: "cloud_id:..."
          # either api_key or username/password, username/password takes precedence if both are included
          # api_key: "..."
          username: "..."
          password: "..."
          es_datastream_name: "logs-generic-default"

and installation is failing:

Screenshot 2024-07-01 at 15 35 26

View stack event:

Screenshot 2024-07-01 at 15 57 33

cc @constanca-m

constanca-m commented 3 months ago

I have tried to deploy it. I updated the needed variables:

In my case, it would be: s3://constanca-test-esf/config.yaml, and this bucket looks like this: image

The config.yaml looks like this:

"inputs":
- "id": "arn:aws:logs:eu-north-1:627286350134:log-group:constanca-test-esf:*"
  "outputs":
  - "args":
      "api_key": "..."
      "elasticsearch_url": "https://terraform-8b3bac.es.eu-central-1.aws.cloud.es.io"
      "es_datastream_name": "logs-esf.cloudwatch-default"
    "type": "elasticsearch"
  "type": "cloudwatch-logs"

Then I sent an event in my cloudwatch logs group: image

And if I look in Discover I can see it there: image

tetianakravchenko commented 3 months ago

So the issue is related to fact that I was trying to deploy lambda application in the region where it was already deployed.

To sum up - we need to update documentation with the information:

tetianakravchenko commented 2 months ago

Also to check: Is it possible to install multiple installations per region? maybe it is possible to fix somehow?

girodav commented 2 months ago

Also to check: Is it possible to install multiple installations per region? maybe it is possible to fix somehow?

It's never been possible (with SAR) due to limitations on the AWS side (full context from Andrea: https://github.com/elastic/elastic-serverless-forwarder/issues/150#issuecomment-1206405134 ). Things might have changed since the last time either myself or Andrea looked at this, so it's worth having a look.

kaiyan-sheng commented 1 month ago

Doc update: https://github.com/elastic/elastic-serverless-forwarder/pull/767