zmoog / public-notes

Apache License 2.0
0 stars 1 forks source link

Figure out how to deploy ESF (AWS console install method) #31

Closed zmoog closed 3 months ago

zmoog commented 1 year ago

Given ESF was published as a private SAR application [^1], I want to install it from the AWS console.

[^1]: See #15 for instructions on how to do it

zmoog commented 1 year ago

Requirements

  1. One SQS queue
  2. One S3 bucket for configuration files

(1) SQS queue

We need an SQS queue for the S3 object creation notifications.

Create a new SQS queue named mbranca-elastic-serverless-forwarder-access-logs and set the visibility timeout to 910 seconds.

(2) S3 bucket

Create a file named mbranca-elastic-serverless-forwarder-configs

And upload a basic configuration file like this:

inputs:
  - type: "s3-sqs"
    id: "arn:aws:sqs:eu-west-1:1234567890:mbranca-elastic-serverless-forwarder-access-logs"
    outputs:
      - type: "elasticsearch"
        args:
          # either elasticsearch_url or cloud_id, elasticsearch_url takes precedence
          elasticsearch_url: "<REDACTED>"
          # either api_key or username/password, api_key takes precedence
          api_key: “<REDACTED>"
          es_datastream_name: "logs-generic-default"
          batch_max_actions: 500
          batch_max_bytes: 10485760
          ssl_assert_fingerprint: ""
zmoog commented 1 year ago

Deploy ESF

Visit the AWS web console and:

  1. Open the Lambda service
  2. Select Applications and click on Create application
  3. From Serverless application select Private applications
  4. Select elastic-serverless-forwarder-maurizio

Set the following values:

Setting Value
ElasticServerlessForwarderS3Buckets arn:aws:s3:::zmoog-esf-data
ElasticServerlessForwarderS3ConfigFile s3://mbranca-elastic-serverless-forwarder-configs/config.yml
ElasticServerlessForwarderS3SQSEvents arn:aws:sqs:eu-west-1:1234567890:mbranca-elastic-serverless-forwarder-access-logs