Collect AWS ECS logs but not found in index

[HuJake]

Hi all,

I have stored the ECS log through firelens to a folder in s3 to store the log, But I found that I can't find relevant information from the OpenSearch dashboard.

Have any index can using?

[nakajiak]

Hi, what logs did you store? Application logs come in a variety of formats and are not automatically loaded. This solution parses the FireLens meta information, but does not parse the logs in it.

See https://github.com/aws-samples/siem-on-amazon-opensearch-service/blob/main/docs/configure_siem.md#customizing-the-log-loading-method

[HuJake]

Hi @nakajiak ,

The following is the log of the application I collected, currently collected in S3, At present, there is a problem in that the Lambda trigger cannot be sent to OpenSearch.

If the log format I currently collect is as follows, how can I trigger Lambda to let it generate an index and send the data to OpenSearch for observation?

[XX][production][XXX 1.16.0][2022-10-13T08:36:10.505Z][INFO][::ffff:1.1.1.1][ROLE:xx] GET /version 200 0 {}
[XX][production][XXX 1.16.0][2022-10-13T08:37:10.505Z][INFO][::ffff:1.1.1.1][ROLE:xx] GET /version 200 0 {}
[XX][production][XXX 1.16.0][2022-10-13T08:38:10.505Z][INFO][::ffff:1.1.1.1][ROLE:xx] GET /version 200 0 {}

Thanks for your help. @nakajiak

[HuJake]

Thanks @nakajiak is there any update ：）

[nakajiak]

Hi, have you created user.ini? If yes, could you share it?

You need to configure via_firelens = True in user.ini https://github.com/aws-samples/siem-on-amazon-opensearch-service/blob/main/source/lambda/es_loader/aws.ini#L50

[HuJake]

Hi @nakajiak,

I have put Lambda Functions （aws.ini in aes-siem-es-loader）

Set to the following configuration.

I have redeployed Lambda but found that the relevant index is still not found in the Opensearch index?

Is there a more detailed ECS configuration process? :(

s3_key_ignored =
# s3_keyにマッチしても無視したいログがあれば、それを識別する文字列を指定
# If there is a log that you want to ignore even if it matches s3_key,
# specify a character string to identify it

# For CloudWatch logs
via_cwl = False
# ログ受信 に CloudWatch Logs 経由をしているか、していないか。
# Are logs received via CloudWatdh Logs?

# For Firelens
via_firelens = True
# ログ受信 に Firelens 経由をして、enable-ecs-log-metadata のフィールドが
# 付与されている場合は、True
# Are logs received via Firelens with ecs-log-metadata? If so, True.
# https://docs.aws.amazon.com/ja_jp/AmazonECS/latest/developerguide/using_firelens.html#firelens-taskdef-metadata

# For Container
ignore_container_stderr = False
# コンテナのログで stderr だった場合、無視して、取り込むのは stdout のみとする

# For CloudWatch logs
loggroup = dummy
# CloudWatch => Kinesis Data Stream 経由でログを受信する時に指定
# Specify string when receiving logs via Kinesis Data Stream via CloudWatch Logs

[HuJake]

Thanks @nakajiak is there any update :-)

[nakajiak]

Hello, please check the aes-siem-es-loader logs from CloudWatch logs. If you have error logs, please share it. Also, can you copy the corresponding setting file in user.ini?

e.g.

[foobar]
index = log-app-foobar
via_firelens = True
s3_key = foobar
file_format = text
timestamp_key = datetime
timestamp_format = iso8601
log_pattern = \[XX\]\[(?P<production>[^ ]+)\].....

[HuJake]

@nakajiak

Below is my relevant settings file. and the error log message

[ERROR] ValueError: there is no valid timestamp_key for index-ecslog
Traceback (most recent call last):
  File "/var/task/aws_lambda_powertools/metrics/metrics.py", line 184, in decorate
    response = lambda_handler(event, context)
  File "/var/task/aws_lambda_powertools/logging/logger.py", line 354, in decorate
    return lambda_handler(event, context, *args, **kwargs)
  File "/var/task/index.py", line 311, in decorator
    return func(*args, **kwargs)
  File "/var/task/index.py", line 347, in lambda_handler
    main(event, context)
  File "/var/task/index.py", line 358, in main
    process_record(record)
  File "/var/task/index.py", line 409, in process_record
    bulkloads_into_opensearch(es_entries, collected_metrics))
  File "/var/task/index.py", line 229, in bulkloads_into_opensearch
    for data in es_entries:
  File "/var/task/index.py", line 174, in get_es_entries
    logparser(lograw, logdata, logmeta)
  File "/var/task/siem/__init__.py", line 546, in __call__
    self.__timestamp = self.get_timestamp()
  File "/var/task/siem/__init__.py", line 971, in get_timestamp
    raise ValueError(msg)```

user.ini is

[index-ecslog]
index_name = log-aws-ecs-app
timestamp_key = date_time
timestamp_format = iso8601
via_firelens = True
log_pattern = \[.*\]\[(?P<staging>[^ ]+)\]\[.* \d.\d\d.\d.\[.*
s3_key = /KDS-S3-ECS-ej2KU/*
ecs = destination.ip
    destination.port
    error.code
    event.code
    event.module
    host.name
    related.ip
    related.user
    source.ip
    source.port
    user.domain
    user.id
    user.name
file_format = text

[nakajiak]

Hello, your regex cannot extract the timestamp correctly.

log_pattern = \[.*\]\[(?P<staging>[^ ]+)\]\[.* \d.\d\d.\d.\[(?P<date_time>[0-9T:.-]*Z)\]

We recommend that you check if you can extract the regular expression from the logs.　https://regex101.com

[nakajiak]

See also this site. https://catalog.us-east-1.prod.workshops.aws/workshops/60a6ee4e-e32d-42f5-bd9b-4a2f7c135a72/en-US/05-ingest-and-process-application-logs/05-5-parse-logs-regex

[HuJake]

@nakajiak

!!! It worked !!!

Thank you very much for your help.

Help me solve many doubts~

[nakajiak]

Nice!