Refresh ECS, Beats manifest and schemas

[shashank-elastic]

Pull Request

Issue link(s): Monthly Refresh as part of Release https://github.com/elastic/ia-trade-team/issues/430

Summary - What I changed

• Monthly Cadence Refresh ECS, Beats and Integration manifest and schemas
• Followed Steps Schema Refresh steps from
• Edge Case for SentinelOne

  
  ❯ python -m detection_rules dev integrations build-manifests -o --prerelease -i sentinel_one_cloud_funnel

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

loading rules to determine all integration tags loaded sentinel_one_cloud_funnel manifests from the following package versions: ['1.3.0', '1.2.1', '1.2.0', '1.1.0', '1.0.0', '0.14.2', '0.14.1', '0.14.0', '0.13.0', '0.12.0', '0.11.0', '0.10.1', '0.10.0', '0.9.0', '0.7.1', '0.7.0', '0.6.0', '0.5.0', '0.4.0', '0.3.0', '0.2.0', '0.1.0'] final integrations manifests dumped: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/integration-manifests.json.gz (.venv) detection-rules on  refresh_schema [$!?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv) on ☁️ shashank.suryanarayana@elastic.co ❯ python -m detection_rules dev integrations build-schemas -o -i sentinel_one_cloud_funnel

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Building integration schemas... processing sentinel_one_cloud_funnel final integrations manifests dumped: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/integration-schemas.json.gz

With the ongoing issue with DGA filed mapping we have revert the Integration Schema Refresh! 

## How To Test

- Unit Test to Pass

## Checklist

<!-- Delete any items that are not applicable to this PR. -->

- [x] Added a label for the type of pr: `bug`, `enhancement`, `schema`, `Rule: New`, `Rule: Deprecation`, `Rule: Tuning`, `Hunt: New`, or `Hunt: Tuning` so guidelines can be generated
- [x] Added the `meta:rapid-merge` label if planning to merge within 24 hours
- [ ] Secret and sensitive material has been managed correctly
- [ ] Automated testing was updated or added to match the most common scenarios
- [ ] Documentation and comments were added for features that require explanation

## Contributor checklist

- Have you signed the [contributor license agreement](https://www.elastic.co/contributor-agreement)?
- Have you followed the [contributor guidelines](https://github.com/elastic/detection-rules/blob/main/CONTRIBUTING.md)?

[protectionsmachine]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a new schema feature to the code.

Documentation and Context

• [ ] Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• [ ] Include additional context or screenshots.
• [ ] Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• [ ] Code follows established design patterns within the repo and avoids duplication.
• [ ] Code changes do not introduce new warnings or errors.
• [ ] Variables and functions are well-named and descriptive.
• [ ] Any unnecessary / commented-out code is removed.
• [ ] Ensure that the code is modular and reusable where applicable.
• [ ] Check for proper exception handling and messaging.

Testing

• [ ] New unit tests have been added to cover the enhancement.
• [ ] Existing unit tests have been updated to reflect the changes.
• [ ] Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• [ ] Validate that any rules affected by the enhancement are correctly updated.
• [ ] Ensure that performance is not negatively impacted by the changes.
• [ ] Verify that any release artifacts are properly generated and tested.

Additional Schema Related Checks

• [ ] Ensure that the enhancement does not break existing functionality. (e.g., run make test-cli)
• [ ] Review the enhancement with a peer or team member for additional insights.
• [ ] Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• [ ] Confirm that all dependencies are up-to-date and compatible with the changes.
• [ ] Link to the relevant Kibana PR or issue provided
• [ ] Exported detection rule(s) from Kibana to showcase the feature(s)
• [ ] Converted the exported ndjson file(s) to toml in the detection-rules repo
• [ ] Re-exported the toml rule(s) to ndjson and re-imported into Kibana
• [ ] Updated necessary unit tests to accommodate the feature
• [ ] Applied min_compat restrictions to limit the feature to a specified minimum stack version
• [ ] Executed all unit tests locally with a test toml rule to confirm passing
• [ ] Included Kibana PR implementer as an optional reviewer for insights on the feature
• [ ] Implemented requisite downgrade functionality
• [ ] Cross-referenced the feature with product documentation for consistency
• [ ] Incorporated a comprehensive test rule in unit tests for full schema coverage
• [ ] Conducted system testing, including fleet, import, and create APIs (e.g., run make test-remote-cli)

[terrancedejesus]

Left a few questions. Did we attempt to refresh MITRE ATT&CK mapping?

[shashank-elastic]

Left a few questions. Did we attempt to refresh MITRE ATT&CK mapping?

We dont have any latest versions for MITRE - https://attack.mitre.org/resources/versions/

[shashank-elastic]

For the Unit Test Failures Debugging

• The Failure is

  Error in both stack and integrations checks: {'stack': KqlParseError('Error at line:1,column:1\nUnknown field\nml_is_dga.malicious_prediction:1 and dns.question.registered_domain:avsvmcloud.com\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nstack: 8.16.0, beats: 8.15.0, ecs: 8.11.0'), 'integrations': KqlParseError("Error at line:1,column:1\nUnknown field\nml_is_dga.malicious_prediction:1 and dns.question.registered_domain:avsvmcloud.com\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n\tTry adding event.module or event.dataset to specify integration module\n\tWill check against integrations ['dga', 'endpoint', 'network_traffic'] combined.\n\tpackage='network_traffic', integration=None, integration_schema_data['package_version']='1.31.0', integration_schema_data['stack_version']='8.10.0', integration_schema_data['ecs_version']='8.10.0'")}

Rule in Question : https://github.com/elastic/detection-rules/blob/main/rules/integrations/dga/command_and_control_ml_dga_activity_using_sunburst_domain.toml

Query

ml_is_dga.malicious_prediction:1 and dns.question.registered_domain:avsvmcloud.com

When we load the schemas these fields are present

>>> from detection_rules.integrations import load_integrations_manifests, load_integrations_schemas
>>> schemas = load_integrations_schemas()
>>> 'dns.question.registered_domain' in schemas['network_traffic']['1.9.3']['dns'].keys()
True
>>> schemas['dga']['2.0.2']
{'dga-2.0.2': {'data_stream.type': 'constant_keyword', 'data_stream.dataset': 'constant_keyword', 'data_stream.namespace': 'constant_keyword', '@timestamp': 'date', 'ml_is_dga.malicious_prediction': 'long', 'ml_is_dga.malicious_probability': 'float'}, 'jobs': ['dga_high_sum_probability']}
>>> 

DAC has touched rule_validators, so initial suspect would be to debug from there

cc @eric-forte-elastic

[shashank-elastic]

Interestingly we have a new DGA version 2.0.3

And the field have massively changed for 2.0.3

{
    "1.0.0": {
        "jobs": [
            "dga_high_sum_probability"
        ]
    },
    "1.0.1": {
        "jobs": [
            "dga_high_sum_probability"
        ]
    },
    "1.1.0": {
        "jobs": [
            "dga_high_sum_probability"
        ]
    },
    "2.0.0": {
        "dga-2.0.0": {
            "data_stream.type": "constant_keyword",
            "data_stream.dataset": "constant_keyword",
            "data_stream.namespace": "constant_keyword",
            "@timestamp": "date",
            "ml_is_dga.malicious_prediction": "long",
            "ml_is_dga.malicious_probability": "float"
        },
        "jobs": [
            "dga_high_sum_probability"
        ]
    },
    "2.0.1": {
        "dga-2.0.1": {
            "data_stream.type": "constant_keyword",
            "data_stream.dataset": "constant_keyword",
            "data_stream.namespace": "constant_keyword",
            "@timestamp": "date",
            "ml_is_dga.malicious_prediction": "long",
            "ml_is_dga.malicious_probability": "float"
        },
        "jobs": [
            "dga_high_sum_probability"
        ]
    },
    "2.0.2": {
        "dga-2.0.2": {
            "data_stream.type": "constant_keyword",
            "data_stream.dataset": "constant_keyword",
            "data_stream.namespace": "constant_keyword",
            "@timestamp": "date",
            "ml_is_dga.malicious_prediction": "long",
            "ml_is_dga.malicious_probability": "float"
        },
        "jobs": [
            "dga_high_sum_probability"
        ]
    },
    "2.0.3": {
        "jobs": [
            "dga_high_sum_probability"
        ]
    }
}

[shashank-elastic]

The issue is changes in upstream Refer - https://github.com/elastic/integrations/pull/10476

• The field definitions have been removed and our validation looks for those fields
• The unit test breaks as these fields are not generated when we build our integration schema dynamically

• Reaching out to sec-ml