lambda-promtail: ALB connection logs are not properly recognized by the `defaultFilenameRegex` expression

[trc-ikeskin]

Describe the bug

AWS released connection logging for Application Load Balancers alongside their new mTLS functionality: https://aws.amazon.com/blogs/aws/mutual-authentication-for-application-load-balancer-to-reliably-verify-certificate-based-client-identities/ in November 2023.

When one activates those logs they are shipped to the ELB log bucket alongside the already existing access logs. However connection log files have a different file name structure than access logs where they are prefixed with the string "connlog".

⚠️ Unfortunately the AWS documentation states the prefix ends with a period, however in reality an underscore is used!

The defaultFilenameRegex does not cover those logs currently which is why below error is thrown:

level=error caller=main.go:269 err="error processing event: type of S3 event could not be determined for object \"AWSLogs/347414xxxxxx/elasticloadbalancing/eu-central-1/2024/02/07/conn_log_347414xxxxxx_elasticloadbalancing_eu-central-1_app.xxxxxxxxx.5ede558f4b0e97a9_20240207T0955Z_10.43.xxx.xxx_b38hdhc6.log.gz\""

https://github.com/grafana/loki/blob/46c6118eb2a2d56406042431dfbed288df6e924b/tools/lambda-promtail/lambda-promtail/s3.go#L78

To Reproduce Steps to reproduce the behavior:

1. Enable connection logs on an Application Load Balancer.
2. Start scraping ELB logs from the referenced bucket using lambda-promtail.

Expected behavior Connection log files should be recognized as valid s3_lb log files and handled by the parser. This could be achieve by adjusting the regex like so: AWSLogs\/(?P<account_id>\d+)\/(?P<type>[a-zA-Z0-9_\-]+)\/(?P<region>[\w-]+)\/(?P<year>\d+)\/(?P<month>\d+)\/(?P<day>\d+)\/(?:conn_log_)?\d+\_(?:elasticloadbalancing|vpcflowlogs)\_\w+-\w+-\d_(?:(?P<lb_type>app|net)\.*?)?(?P<src>[a-zA-Z0-9\-]+)

Environment:

• Infrastructure: AWS, AWS Lambda, AWS S3
• Deployment tool: Terraform

Screenshots, Promtail config, or terminal output

level=error caller=main.go:269 err="error processing event: type of S3 event could not be determined for object \"AWSLogs/347414xxxxxx/elasticloadbalancing/eu-central-1/2024/02/07/conn_log_347414xxxxxx_elasticloadbalancing_eu-central-1_app.xxxxxxxxx.5ede558f4b0e97a9_20240207T0955Z_10.43.xxx.xxx_b38hdhc6.log.gz\""

[trc-ikeskin]

On a second thought it might be a good idea to add new log types such as s3_lb_access and s3_lb_connection since the differ a bit in format and further upstream use (e.g. in Grafana) would become easier.

[JEFFCTSAI]

I'm current trying to forward my AWS MSK logs from s3 to Loki using lambda-promtail and also running into this issue

level=error caller=main.go:236 err="error processing event: type of S3 event could not be determined for object \"aws_logs/AWSLogs/<account number>/KafkaBrokerLogs/us-east-1/msk-nonprod-*****/2024-09-30-22/Broker-1_22-05_***.log.gz\""

are we limited to only allow certain types of logs in s3 to be sent to Loki using this lambda?