Invalid regex pattern of alb

[mrobinson1022]

ALB access logs have been ingesting into opensearch using the es-loader function for a while now. This morning they appear to have completely stopped due to a mismatch on the regex pattern.

I'm seeing this in the logs:

{
    "level": "ERROR",
    "message": "Invalid regex pattern of alb",
    "location": "convert_lograw_to_dict:70",
    "timestamp": "2023-10-06 14:05:08,603+0000",
    "service": "es-loader",
    "cold_start": false,
    "function_name": "aes-siem-es-loader",
    "function_memory_size": "2048",
    "function_arn": "arn:aws:lambda:us-east-1:<os_account_number>:function:aes-siem-es-loader",
    "function_request_id": "97134df9-54e7-4b06-9299-70843577211f",
    "s3_key": "AWSLogs/<lb_account_number>/elasticloadbalancing/us-east-1/2023/10/06/<lb_account_number>_elasticloadbalancing_us-east-1_app.<application_name>-load-balancer.70da16283c7eda86_20231006T1405Z_192.168.1.100_4bpu49ue.log.gz",
    "s3_bucket": "aes-siem-<os_account_number>-log",
    "message_rawdata": "https 2023-10-06T14:00:59.248747Z app/<application_name>-load-balancer/70da16283c7eda86 167.96.123.45:52200 10.33.195.230:80 0.000 0.008 0.000 200 200 42 5649 \"GET https://192.168.1.100:443/ HTTP/1.1\" \"-\" TLS_AES_128_GCM_SHA256 TLSv1.3 arn:aws:elasticloadbalancing:us-east-1:<lb_account_number>:targetgroup/<application_name>-group/628c165c575e7a7e \"Root=1-6520131b-540d1be4066ae74f1bbcb40a\" \"-\" \"arn:aws:acm:us-east-1:<lb_account_number>:certificate/7d5f2b2a-f6aa-4b3e-a924-ffa443662c60\" 0 2023-10-06T14:00:59.240000Z \"forward\" \"-\" \"-\" \"192.168.1.200:80\" \"200\" \"-\" \"-\"",
    "message_regex_pattern": "re.compile('(?P<request_type>[^ ]*) (?P<timestamp>[^ ]*) (?P<elb>[^ ]*) (?P<client_ip>[^ ]*):(?P<client_port>[0-9]*) (?P<target_ip>[^ ]*)[:-](?P<target_port>[0-9]*) (?P<request_processing_time>[-.0-9]*) (?P<targ)",
    "xray_trace_id": "1-65201414-1239605b5038d22a3bfe456e"
}

Any guidance you can give would be greatly appreciated!

[mrobinson1022]

it appears a rouge agent (a team member) disconnected this ALB from the logs bucket, which is why we suddenly had zero logs. So it seems a majority of the ELB logs are being ingested, however some are still facing this issue. I'm curious of the cause, but the main issue of NO logs at all has been solved 🤣

[nakajiak]

What SIEM version are you using?

[nakajiak]

ssl_cipher of logs seems changed

your sample log: ssl_cipher uses underscores

TLS_AES_128_GCM_SHA256 TLSv1.3

On the other hand, the sample log in the official documentation uses dashes.

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html

[nakajiak]

workaround: Edit part of log_pattern in the [alb] section of aws.ini

Modification place

From:

(?P<ssl_cipher>[()A-Z0-9-]+)

To:

(?P<ssl_cipher>[()A-Z0-9_-]+)

https://github.com/aws-samples/siem-on-amazon-opensearch-service/blob/v2.10.2/source/lambda/es_loader/aws.ini#L801

[nakajiak]

Or please update SIEM version to v2.10.2a

[mrobinson1022]

Thanks so much for spotting that @nakajiak! It looks like in typical aws fashion, three of the TLS ciphers use underscores rather than dashes.

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html

[nakajiak]

I didn't know the three TLS ciphers. Thanks so much.