[AWS][Cloudfront Logs] - Implemented GROK processor based ipv6/v4 parsing in AWS Cloudfront Logs data stream

ShourieG commented 4 days ago

Type of change

Please label this PR with one of the following labels, depending on the scope of your change:

Bug
enhancement

Proposed commit message

WHAT: Implemented GROK processor based ipv6/v4 parsing. Cleaned up formatting issues across all data streams via elastic-package format.
WHY: The regex pattern used for the ipv6 was overtly complex and caused errors in Elasticsearch. The cleanup was a byproduct of running elastic-package format.
HOW: A new helper pipeline was added containing this new grok processor and helper scripts which are invoked in a foreach processor from the parent pipeline. This approach had to be taken because with existing limitations of one sub processor per foreach and grok being unable to append to existing lists/arrays.

The Elasticsearch error produced is as follows:-

Processor 'conditional' failed with message '[scripting] Regular expression considered too many characters, pattern: [((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(250-5){3})){3})/)|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(250-5){3})){3})/)|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(250-5){3}))){3}))/)|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(250-5){3}))){3}))/)|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(250-5){3}))){3}))/)|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(250-5){3}))){3}))/)|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(250-5){3}))){3}))/)|:)))(%.+)??$], limit factor: [6], char limit: [114], count: [115], wrapped: [2a06:98c0:3600::103], this limit can be changed by changed by the [script.painless.regex.limit-factor] setting'

NOTE

The latest commit - elastic-packge build cleaned up a lot of formatting issues, hence there are added changes across many data streams. I think these are good in the long run hence kept them in.

Checklist

[x] I have reviewed tips for building integrations and this pull request is aligned with them.
[x] I have verified that all data streams collect metrics or logs.
[x] I have added an entry to my package's changelog.yml file.
[x] I have verified that Kibana version constraints are current according to guidelines.
[ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

elasticmachine commented 4 days ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy commented 4 days ago

@ShourieG , would be it possible to replace the script processor with foreach and grok with on_failure to use the default IPV6 pattern provided by the logstash?

ShourieG commented 4 days ago

would be it possible to replace the script processor with foreach and grok with on_failure to use the default IPV6 pattern provided by the logstash?

@kcreddy, should we try it for both ipv4/v6 or only ipv6 and leave ipv4 as it is currently ?

kcreddy commented 4 days ago

@kcreddy, should we try it for both ipv4/v6 or only ipv6 and leave ipv4 as it is currently ?

Makes sense to try for both and use default IPV4 and IPV6 pattern provided. Example in use. It also helps to keep the pattern in sync in case of any future changes to it at source.

elastic-vault-github-plugin-prod[bot] commented 4 days ago

:rocket: Benchmarks report

To see the full report comment with /test benchmark fullreport

ShourieG commented 3 days ago

@kcreddy, have updated the logic with a relevant GROK processor but still some scripts were required for cleanup and handling the localhost edge-case scenario. There are some removals in the ip section based on the GROK patterns.

agithomas commented 1 day ago

LGTM!

Just a suggestion: Can we consider adding a sample or samples to the pipeline test to capture the pattern observed in the reported issue, handling of localhost etc?

elasticmachine commented 1 day ago

:green_heart: Build Succeeded

Buildkite Build
Commit: f7ae0bba7e9943fce9761be3524dc0973cdab818

History

:green_heart: Build #18634 succeeded 3328d815eb6cfa58549b89b82a7206ec3069a8c9
:green_heart: Build #18613 succeeded 7d36c97b9d1ab3f47b32270cd23ad378217191fd
:broken_heart: Build #18609 failed dd3e19759c44a5c9a45f17e9bf500af71c44bb78
:green_heart: Build #18603 succeeded 3cf60a7f3d4bb50b415a91fe50e1b98bbb8d6df1

cc @ShourieG

elastic-sonarqube[bot] commented 1 day ago

Quality Gate failed

Failed conditions
74.6% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

ShourieG commented 1 day ago

LGTM!

Just a suggestion: Can we consider adding a sample or samples to the pipeline test to capture the pattern observed in the reported issue, handling of localhost etc?

Current tests already contain the localhost pattern @agithomas. The reported issue on the other hand was reporting errors with the existing regex pattern when compiling in elasticsearch.

elastic-vault-github-plugin-prod[bot] commented 1 day ago

Package aws - 2.32.0 containing this change is available at https://epr.elastic.co/package/aws/2.32.0/

elastic / integrations