Fix errors: data external, use resources arns if they exist, point correctly to config.yaml

Description

This PR is the pickup from this PR: https://github.com/elastic/terraform-elastic-esf/pull/1.

The linked PR does the following:

Fixes the error reported in issue https://github.com/elastic/terraform-elastic-esf/issues/3 by iterating over the resources (kinesis data stream, sqs, s3) arns only if they exist
Removes use of data "external" in the esf.tf file

However, even after these changes, there were still issues with the files.

So this PR also makes the following changes:

Change 1:

The mentions to the config file were incorrect. It only pointed to the bucket. It failed with:

[ERROR] ConfigFileException: Invalid s3 uri provided: `s3://constanca-benchmark-test-esf-bucket`
Traceback (most recent call last):
  File "/var/task/handlers/aws/utils.py", line 63, in wrapper
    return func(lambda_event, lambda_context)
  File "/var/task/handlers/aws/utils.py", line 98, in wrapper
    raise e
  File "/var/task/handlers/aws/utils.py", line 82, in wrapper
    return func(lambda_event, lambda_context)
  File "/var/task/handlers/aws/handler.py", line 67, in lambda_handler
    raise ConfigFileException(e)

We need to add to update the environment variable S3_CONFIG_FILE to point to the right file:

S3_CONFIG_FILE : "${local.s3-url-config-file}/config.yaml"

And also update in the permissions to avoid access denied:

config-file = {
    effect    = "Allow",
    actions   = ["s3:GetObject"],
    resources = ["${var.config-file-bucket}/config.yaml"]
},

Note: The file needs to be necessarily named config.yaml if we set it this way. It does not seem like a problem since we mention it in the documentation as well:

Elastic Serverless Forwarder requires a config.yaml file to be uploaded to an S3 bucket and referenced by the S3_CONFIG_FILE environment variable.

Change 2:

Data "external" was replaced with data "null_data_source", but that is deprecated:

│ Warning: Deprecated
│ 
│   with data.null_data_source.esf-source-path,
│   on esf.tf line 68, in data "null_data_source" "esf-source-path":
│   68: data "null_data_source" "esf-source-path" {
│ 
│ The null_data_source was historically used to construct intermediate values to re-use elsewhere in configuration, the same can now be achieved using locals or the terraform_data resource type in Terraform 1.4 and later.
│ 
│ (and one more similar warning elsewhere)
╵

Replace it with a local variable, and make the module esf-lambda-function depend on the ESF source download so it does not cause issues with a source_path that does not exist.

Issues

Closes https://github.com/elastic/terraform-elastic-esf/issues/3.

Results

(Pre requisites for this: have a cloudwatch log group and S3 bucket with config.yaml file)

I tested this using a cloudwatch logs input. Example:

inputs:
  - id: "arn:aws:logs:eu-central-1:627286350134:log-group:constanca-benchmark-test-cloudwatch-lg:*"
    outputs:
      - args:
          api_key: "<Api Key>"
          elasticsearch_url: "<url>"
          es_datastream_name: "logs-generic-default"
        type: "elasticsearch"
    type: "cloudwatch-logs"

This file was in my S3 bucket, and I set value for variables:

lambda-name = "constanca-lambda-tf-esf"
release-version= "lambda-v1.8.0"
config-file = "arn:aws:s3:::constanca-benchmark-test-esf-bucket"
aws_region = "eu-central-1"
cloudwatch-logs = ["arn:aws:logs:eu-central-1:627286350134:log-group:constanca-benchmark-test-cloudwatch-lg:*"]

Results in Discover:

To do next

We still need to remove dependencies, but this PR just tries to get this repository in a functional state.

elastic / terraform-elastic-esf