Figure out how to deploy ESF (script install method)

[zmoog]

I need to deploy ESF to run some tests, but this is the first time I have done it.

I know there is a there is Deploy Elastic Serverless Forwarder page, so let's start from here.

[zmoog]

I guess the Deploy Elastic Serverless Forwarder directly section is what I was looking for.

[zmoog]

(1) Create and upload config.yaml to S3 bucket

I have a bucket called zmoog-esf-config already available with the following content:

inputs:
  - type: "s3-sqs"
    id: "arn:aws:sqs:eu-west-1:123:whatever-access-logs"
    outputs:
      - type: "elasticsearch"
        args:
          api_key: "abcdef=="
          es_datastream_name: "logs-generic-default"
          batch_max_actions: 500
          batch_max_bytes: 10485760
          ssl_assert_fingerprint: ""

(2) Create publish-config.yaml for the publishing script

To deploy the forwarder directly, you need to define a publish-config.yaml file and pass this as an argument in the publishing script.

Save the following YAML content as publish-config.yaml and edit as required before running the publishing script. You should remove any inputs or arguments you are not using.

kinesis-data-stream:
    - arn: "arn:aws:kinesis:%REGION%:%ACCOUNT%:stream/%STREAMNAME%"
      batch_size: 10
      batching_window_in_second: 0
      starting_position: TRIM_HORIZON
      starting_position_timestamp: 0
      parallelization_factor: 1
sqs:
    - arn: "arn:aws:sqs:%REGION%:%ACCOUNT%:%QUEUENAME%"
      batch_size: 10
      batching_window_in_second: 0
s3-sqs:
    - arn: "arn:aws:sqs:%REGION%:%ACCOUNT%:%QUEUENAME%"
      batch_size: 10
      batching_window_in_second: 0
cloudwatch-logs:
    - arn: "arn:aws:logs:%AWS_REGION%:%AWS_ACCOUNT_ID%:log-group:%LOG_GROUP_NAME%:*"
    - arn: "arn:aws:logs:%AWS_REGION%:%AWS_ACCOUNT_ID%:log-group:%LOG_GROUP_NAME%:log-stream:%LOG_STREAM_NAME%"
ssm-secrets:
  - "arn:aws:secretsmanager:%AWS_REGION%:%AWS_ACCOUNT_ID%:secret:%SECRET_NAME%"
kms-keys:
    - "arn:aws:kms:%AWS_REGION%:%AWS_ACCOUNT_ID%:key/%KMS_KEY_UUID%"
s3-buckets:
    - "arn:aws:s3:::%BUCKET_NAME%"
subnets:
    - "%SUBNET_ID%"
security-groups:
    - "%SECURITY_ID%"
s3-config-file: "s3://%S3_CONFIG_BUCKET_NAME%/%S3_CONFIG_OBJECT_KEY%"
continuing-queue:
    batch_size: 10
    batching_window_in_second: 0

I'll keep the following pieces replacing the placeholder with the actual values:

s3-sqs:
    - arn: "arn:aws:sqs:eu-west-1:123456:whatever-access-logs"
      batch_size: 10
      batching_window_in_second: 0
s3-buckets:
    - "arn:aws:s3:::whatever-access-logs"
s3-config-file: "s3://whatever-elastic-serverless-forwarder-config/config.yml"
continuing-queue:
    batch_size: 10
    batching_window_in_second: 0

[zmoog]

This install method requires an additional depenency:

$ pip3 install awscli aws-sam-cli ruamel.yaml

Ready to go:

$ ./publish_lambda.sh \
    publish-config.yaml \
    forwarder-lambda \
    telemetry \
    zmoog-elastic-serverless-forwarder-artifacts \
    eu-west-1

[zmoog]

Note: I had to update the source SQS queue visibility from the default value (30s) to the lambda timeout (910s).

[zmoog]

I also had to change the git repo URL to deploy from my own fork:

diff --git a/publish_lambda.sh b/publish_lambda.sh
index 5f25770..26d6215 100755
--- a/publish_lambda.sh
+++ b/publish_lambda.sh
@@ -31,7 +31,8 @@ REGION="$5"
 TMPDIR=$(mktemp -d /tmp/publish.XXXXXXXXXX)
 CLONED_FOLDER="${TMPDIR}/sources"
 PACKAGE_FOLDER="${CLONED_FOLDER}/package"
-GIT_REPO="https://github.com/elastic/elastic-serverless-forwarder.git"
+# GIT_REPO="https://github.com/elastic/elastic-serverless-forwarder.git"
+GIT_REPO="https://github.com/zmoog/elastic-serverless-forwarder.git"

 trap 'rm -rf ${TMPDIR}' EXIT

[zmoog]

After the first deploy, you can make changes to the publish-config.yaml file or the application code and deploy it again running the same command:

$ ./publish_lambda.sh \
    publish-config.yaml \
    forwarder-lambda \
    telemetry \
    zmoog-elastic-serverless-forwarder-artifacts \
    eu-west-1

[zmoog]

Improvement: instead of using the data stream logs-generic-default we can switch to logs-aws.s3access-default to take advantage of the ingest pipeline:

inputs:
  - type: "s3-sqs"
    id: "arn:aws:sqs:eu-west-1:123456:whatever-dev-access-logs" 
    outputs:
      - type: "elasticsearch"
        args:
          es_datastream_name: "logs-aws.s3access-default"
          batch_max_actions: 500
          batch_max_bytes: 10485760
          ssl_assert_fingerprint: ""

[zmoog]

Pro tip: if you make a manual change to the Lambda using the web console (eg. adding an environment variable) it will be lost in the following deploy.