Security Lake custom source configuration for es-loader

[chrisammon3000]

The doc for Security Lake integration says that if the S3 object name doesn't match the current config it has to be reconfigured in user.ini:

[securitylake]
s3_key = [0-9a-f]{32}\.gz\.parquet|[Ss]ecurity[Ll]ake/|file_name_of_parquet

I have set mine as follows:

[securitylake]
s3_key = [0-9a-f]{32}\.gz\.parquet|[Ss]ecurity[Ll]ake/|[a-zA-Z0-9-]+\.gz

in order to match these S3 objects delivered from Security Lake custom sources:

ext/OCSF_LAMBDA/region=us-east-1/accountId=1234567891011/eventday=20230811/dev-sl-ocsf-processing-OcsfLambdaLogsCustomSourceDe-GjVRhffh11x3-2-2023-08-11-22-16-56-f446a497-81b0-3e44-931c-c8fc297abcd1.gz
ext/OCSF_WAF/region=us-east-1/accountId=1234567891011/eventday=20230811/dev-sl-ocsf-processing-OcsfWafLogsCustomSourceDeliv-1ECAj4uSWnt6-2-2023-08-11-22-10-59-3a8d80f4-a649-33dc-a794-ac6d733ed35a.gz
ext/OCSF_WAF/region=us-east-1/accountId=1234567891011/eventday=20230811/dev-sl-ocsf-processing-OcsfWafLogsCustomSourceDeliv-1ECAj4uSWnt6-2-2023-08-11-22-16-02-62d1719f-977a-3422-96a1-61017a777c85.gz
ext/OCSF_WAF/region=us-east-1/accountId=1234567891011/eventday=20230811/dev-sl-ocsf-processing-OcsfWafLogsCustomSourceDeliv-1ECAj4uSWnt6-2-2023-08-11-22-16-42-22b6a235-5e29-3fdc-9e95-aeb8d4aa538c.gz

Does not appear to work because I am not finding the records in the ES. How should I configure user.ini so that es-loader will find these objects and load them to ES?

[nakajiak]

I have checked the regex by https://regex101.com/ . It seems good. Can you take a look at the SIEM dashboard in CloudWatch Dashbaord? Maybe you can find the error log or something else. If you still can't figure it out, please let us know the processing message of the parquet file in /aws/lambda/aes-siem-es-loader in CloudWatch Logs.

[chrisammon3000]

Does it matter that I'm using gzip JSON from Firehose instead of parquet?

EDIT: Switched to parquet format, will see if it works. This is the new config:

[securitylake]
s3_key = [0-9a-f]{32}\.gz\.parquet\|[Ss]ecurity[Ll]ake/|[a-zA-Z0-9-]+\.parquet

For these files:

ext/OCSF_LAMBDA/region=us-east-1/accountId=138716245237/eventday=20230821/dev-sl-ocsf-processing-OcsfLambdaLogsCustomSourceDe-GjVRhffh11x3-3-2023-08-21-17-36-54-63fc74e4-1c17-3ab0-a8b7-2c9ea02ad63f.parquet
ext/OCSF_LAMBDA/region=us-east-1/accountId=138716245237/eventday=20230821/dev-sl-ocsf-processing-OcsfLambdaLogsCustomSourceDe-GjVRhffh11x3-3-2023-08-21-17-41-03-f7169288-2a56-35f8-9b6a-d62b9a81d7a2.parquet
ext/OCSF_LAMBDA/region=us-east-1/accountId=138716245237/eventday=20230821/dev-sl-ocsf-processing-OcsfLambdaLogsCustomSourceDe-GjVRhffh11x3-3-2023-08-21-17-42-03-f21c0c6d-eb90-36b4-b9ce-c010a0bc9f0b.parquet
ext/OCSF_WAF/region=us-east-1/accountId=138716245237/eventday=20230821/dev-sl-ocsf-processing-OcsfWafLogsCustomSourceDeliv-1ECAj4uSWnt6-3-2023-08-21-17-30-56-16a6d5bc-1e75-3389-ac54-115e712355a5.parquet

[chrisammon3000]

Incidentally, it looks like a separate and probably unrelated issue might be occurring that probably requires it's own issue. Please see this stack trace from the es-loader logs, looks related to Lambda PowerTools:

[ERROR] RuntimeError: No active exception to reraise
Traceback (most recent call last):
  File "/var/task/aws_lambda_powertools/metrics/metrics.py", line 184, in decorate
    response = lambda_handler(event, context)
  File "/var/task/aws_lambda_powertools/logging/logger.py", line 354, in decorate
    return lambda_handler(event, context, *args, **kwargs)
  File "/var/task/index.py", line 397, in decorator
    return func(*args, **kwargs)
  File "/var/task/index.py", line 464, in lambda_handler
    main(event, context)
  File "/var/task/index.py", line 491, in main
    process_record(recs)
  File "/var/task/index.py", line 546, in process_record
    raise

[chrisammon3000]

Figured it out, I believe the culprit was not having the custom sources selected under the Security Lake subscription. I can update the documentation and create a pull request for this, but the issue is resolved so I'll close it.