esf-terraform

You can find details on ESF in Elastic Serverless Forwarder for AWS.

This repository contains all necessary resources to deploy ESF.

Requirements

• curl
• terraform

How to use

1. Define secrets and variables in *.auto.tfvars files. See variables.tf for the list of variables declared or read section Inputs. Example:

   
   # variables.auto.tfvars

lambda-name = "my-esf-lambda" release-version = "lambda-v1.9.0" # See https://github.com/elastic/elastic-serverless-forwarder/tags

config-file-bucket = "my-esf-bucket" # Uncomment if s3 bucket pre-exists

aws_region = "eu-central-1"

config-file-local-path = "./config.yaml" # Uncomment if local config path is used

inputs = [ { type = "cloudwatch-logs" id = "" outputs = [ { type = "elasticsearch" args = { elasticsearch_url = "https://url.com" api_key = "" es_datastream_name = "logs-esf.cloudwatch-default" } } ] } ]

Please read section [Inputs configuration](#inputs-configuration) for more details on how to configure the inputs.
2. Execute `terraform init`
3. Execute `terraform apply`

## Inputs configuration

> Note: Read [Create and upload config.yaml to S3 bucket](https://www.elastic.co/guide/en/esf/current/aws-deploy-elastic-serverless-forwarder.html#sample-s3-config-file) if you need more details on how the inputs should be configured.

> Note: Read [Fields](https://www.elastic.co/guide/en/esf/current/aws-deploy-elastic-serverless-forwarder.html#s3-config-file-fields) to know which values are expected for each field input.

> Warning: If you use `s3-sqs` input type, you also need to configure `s3-buckets` variable.

When applying these configuration files, a `config.yaml` file will always be uploaded to an S3 bucket. This S3 bucket will be the one specified in `config-file-bucket`, or, if the value is left empty, a new S3 bucket will be created.

Following this, we will create the content for the `config.yaml` file. This file will be built based on:
- Variable `inputs`. This variable is not required.
- Local configuration file found in `config-file-local-path`. This variable is also not required.

If both variables are provided, both will be considered. Otherwise, just the one that was given. If none are provided, the `config.yaml` file will be:

```yaml
"inputs": []

It does not make sense to leave both empty.

You can see the following examples on the resulting config.yaml file.

Configure just the inputs variable

Configure the inputs variable as:

inputs = [
  {
    type = "cloudwatch-logs"
    id   = "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-inputs-variable:*"
    outputs = [
      {
        type = "elasticsearch"
        args = {
          elasticsearch_url  = "<url>"
          api_key            = "<api key>"
          es_datastream_name = "logs-esf.cloudwatch-default"
        }
      }
    ]
  }
]

Do not configure the config-file-bucket variable, which will be left as (empty) since that is the default.

The config.yaml placed inside the bucket will be:

"inputs":
  - "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-inputs-variable:*"
    "outputs":
      - "args":
          "api_key": "<api key>"
          "elasticsearch_url": "<url>"
          "es_datastream_name": "logs-esf.cloudwatch-default"
        "type": "elasticsearch"
    "type": "cloudwatch-logs"

Configure just the config-file-local-path variable

Do not configure the inputs variable, which will be left as [] since that is the default.

Configure config-file-local-path variable:

config-file-local-path = "./config.yaml"

And the local config.yaml file looks like:

"inputs":
  - "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-local-file:*"
    "outputs":
      - "args":
          "api_key": "<api key>"
          "elasticsearch_url": "<url>"
          "es_datastream_name": "logs-esf.cloudwatch-default"
        "type": "elasticsearch"
    "type": "cloudwatch-logs"

Configure both variables

Configure both inputs and config-file-local-path like in the previous examples.

The config.yaml placed inside the bucket will be:

"inputs":
- "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-inputs-variable:*"
  "outputs":
  - "args":
      "api_key": "<api key>"
      "elasticsearch_url": "<url>"
      "es_datastream_name": "logs-esf.cloudwatch-default"
    "type": "elasticsearch"
  "type": "cloudwatch-logs"
- "id": "arn:aws:logs:eu-central-1:627286350134:log-group:coming-from-local-file:*"
  "outputs":
  - "args":
      "api_key": "<api key>"
      "elasticsearch_url": "<url>"
      "es_datastream_name": "logs-esf.cloudwatch-default"
    "type": "elasticsearch"
  "type": "cloudwatch-logs"

Requirements

Providers

Modules

Resources

Inputs

object({
    batch_size                = optional(number, 10)
    batching_window_in_second = optional(number, 0)
  })

{
  "batch_size": 10,
  "batching_window_in_second": 0
}

list(object({
    type = string
    id   = string
    outputs = list(object({
      type = string
      args = object({
        elasticsearch_url      = optional(string)
        logstash_url           = optional(string)
        cloud_id               = optional(string)
        api_key                = optional(string)
        username               = optional(string)
        password               = optional(string)
        es_datastream_name     = string
        batch_max_actions      = optional(number)
        batch_max_bytes        = optional(number)
        ssl_assert_fingerprint = optional(string)
        compression_level      = optional(string)
      })
    }))
  }))

object({
    security-groups = list(string)
    subnets         = list(string)
  })

{
  "security-groups": [],
  "subnets": []
}

Outputs