Add initial support to validate the mappings in system tests

mrodm commented 3 weeks ago

Relates https://github.com/elastic/elastic-package/issues/2206

Add validation of the mappings when running system tests.

This process compares the mappings installed by Fleet (preview mappings) with the ones obtained after ingesting new documents into Elasticsearch.

This validation for the time being is behind an environment variable as a technical preview:

# Default value and same behavior as now (validation of fields)
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=fields elastic-package test system -v

# Test just validation of mappings 
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=mappings elastic-package test system -v

# Test both validation of mappings and fields
ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=all elastic-package test system -v

Assumptions considered while working on this PR: https://github.com/elastic/elastic-package/issues/2206#issuecomment-2468807742

As part of this PR:

CI pipeline has been updated to remove steps related to running system tests with Elastic Agent.
- It will be kept one step to ensure that the feature is working.
CI pipeline has been updated to add steps to run mapping validation in system tests.

Author's checklist

[x] Update CI steps to remove old stack agents
[x] Update CI to add new steps for validating fields using mappings
[x] Add environment variable to validate that fields are documented with different methods
[x] Add new environment variables to docs.
[ ] Remove debug/testing log statements
[x] Test with Elastic stack 8.x
- Some packages are failing like prometheus failing since some fields depend on dynamic templates (not implemented yet)
[x] Test with Elastic stack 7.x (f5 package in integrations repository)
[x] Test with integration and input packages (sql_input test package in elastic-package repo).
[x] Test with Elastic stacks with LogsDB enabled (synthetics).
- Tested in CI with ti_anomali_logsdb and auth0_logsdb

mrodm commented 2 weeks ago

/test

mrodm commented 1 week ago

test integrations

elastic-vault-github-plugin-prod[bot] commented 1 week ago

Created or updated PR in integrations repository to test this version. Check https://github.com/elastic/integrations/pull/11828

mrodm commented 1 week ago

Looking at the integrations PR testing with the new validation mappings (not all packages tested, there were some filtered for now):

there are some packages that fail because they use dynamic templates and they are not supported yet:
- airflow, cockroachdb, docker, etcd, istio, kibana, linux, system, prometheus, prometheus_input, ti_mandiant_advantage
there are packages that looks like they are missing some field definitions:
- auditd_manager: fields: auditd.data.a0, auditd.data.a1, auditd.data.a2, auditd.data.a3
- sql_input: fields: sql.metrics.active_time, sql.metrics.cntr_value...
  - There is a dynamic template that it is not created: https://github.com/elastic/integrations/blob/79ec43e48fdee7752f57fc66b480a5de34c5d45a/packages/sql_input/fields/input.yml#L23
box_events, claroty_ctd, mimecast, ti_anomali:
- failures like for threat.indicator.modified_at different types in ECS and in Elasticsearch
- failures depend on the stack version
cef fails for url.original
- type in ECS is wildcard and the one set in Elasticsearch is text
cisco_ise failures at pipeline tests, not related
gcp: failing a field with type nested that the preview does not contain properties.
- "ip_port_info":{"type":"nested","properties":{"ip_protocol":{"type":"keyword","ignore_above":1024},"port_range":{"type":"keyword","ignore_above":1024}}}
- "ip_port_info":{"type":"nested"}
ibmmq: process.pid float type vs long type in ECS.
m365_defender: missing definition for path process.parent (all its child exists, but that specific field in ECS does not)

cc @jsoriano

mrodm commented 1 week ago

gcp: failing a field with type nested that the preview does not contain properties.

"ip_port_info":{"type":"nested","properties":{"ip_protocol":{"type":"keyword","ignore_above":1024},"port_range":{"type":"keyword","ignore_above":1024}}} "ip_port_info":{"type":"nested"}

This failure has been skipped, as for packages with spec < 3.0.1 it was not checked when using the fields validation and gcp is defining 3.0.0: https://github.com/elastic/elastic-package/blob/8cc126ae5015dd336b22901c365e8c98db4e7c15/internal/fields/validate.go#L1200

mrodm commented 1 week ago

m365_defender: missing definition for path process.parent (all its child exists, but that specific field in ECS does not)

This field was an empty object (but inside another that neither was present in the preview):

  "process": {
    "properties": {
      "parent": {
        "type": "object"
      }
    }
  }

Maybe as a result of the ingest pipeline?

As process does not exist in the preview, it was missing to skip these kind of objects (empty ones) in this case too.

Added the skip logic for this scenario here: https://github.com/elastic/elastic-package/pull/2214/commits/cdf78218e3f351b92b1eaa332f2783d040768664

mrodm commented 1 week ago

test integrations

elastic-vault-github-plugin-prod[bot] commented 1 week ago

Created or updated PR in integrations repository to test this version. Check https://github.com/elastic/integrations/pull/11828

mrodm commented 1 week ago

box_events, claroty_ctd, mimecast, ti_anomali:

failures like for threat.indicator.modified_at different types in ECS and in Elasticsearch

failures depend on the stack version

There is a difference in the ecs@mappings component template between 8.13.0 and 8.15.3 related to the dynamic template to detect dates causing these errors:

8.13.0 (it is missing *.indicator.last_seen, *.indicate.first_seen and *.indicator.modified_at)

    {
      "ecs_date": {
        "path_match": [
          "*.timestamp",
          "*_timestamp",
          "*.not_after",
          "*.not_before",
          "*.accessed",
          "created",
          "*.created",
          "*.installed",
          "*.creation_date",
          "*.ctime",
          "*.mtime",
          "ingested",
          "*.ingested",
          "*.start",
          "*.end"
        ],
        "unmatch_mapping_type": "object",
        "mapping": {
          "type": "date"
        }
      }
    },

8.15.3

    {
      "ecs_date": {
        "path_match": [
          "*.timestamp",
          "*_timestamp",
          "*.not_after",
          "*.not_before",
          "*.accessed",
          "created",
          "*.created",
          "*.installed",
          "*.creation_date",
          "*.ctime",
          "*.mtime",
          "ingested",
          "*.ingested",
          "*.start",
          "*.end",
          "*.indicator.first_seen",
          "*.indicator.last_seen",
          "*.indicator.modified_at",
          "*threat.enrichments.matched.occurred"
        ],
        "unmatch_mapping_type": "object",
        "mapping": {
          "type": "date"
        }
      }
    }

Not sure if it could be applied some kind of exception @jsoriano

elasticmachine commented 1 week ago

:green_heart: Build Succeeded

Buildkite Build
Commit: cdf78218e3f351b92b1eaa332f2783d040768664

History

:green_heart: Build #4359 succeeded f4451d22719ea1af4122ea3894f2ed80910e7588
:green_heart: Build #4355 succeeded e20d56397d12cc33a56a9243b6f246764ec3d60b
:green_heart: Build #4352 succeeded 650b3ed6d02816b90f962a4542dde371cd021268
:green_heart: Build #4351 succeeded a058dfbed0e17d01140373d6074184bb93ce83ad
:green_heart: Build #4338 succeeded 5730b0e697885a69a413bdc337274e862bc764f4
:yellow_heart: Build #4329 was flaky 21dca78c4318b518d34413cd07a950f35a44b8c5

cc @mrodm

mrodm commented 1 week ago

For the cef package there is a similar error as in box_events and others:

8.6.1 dynamic templates (this mapping does not match url.original):

    {
      "_embedded_ecs-url_original_to_multifield": {
        "path_match": "*.url.original",
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        }
      }
    },

8.15.3

    {
      "ecs_path_match_wildcard_and_match_only_text": {
        "path_match": [
          "*.body.content",
          "*url.full",
          "*url.original"
        ],
        "unmatch_mapping_type": "object",
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        }
      }
    },

mrodm commented 1 week ago

In the second build from integrations, teleport started to fail but I could not reproduce it yet locally: https://buildkite.com/elastic/integrations/builds/18608#_ but in the third attempt it run successfully. I don't know why there could be differences in the mappings.

In the first build, this package was not failing: https://buildkite.com/elastic/integrations/builds/18601#_

mrodm commented 1 week ago

The error related to ibmmq it's about process.pid. In this case, this field comes from the Ingest pipeline where it is set the type float (link). This field is not defined in the package, and when checking with ECS it fails because ECS sets for this field long type (link).

This package fails with both 8.14.0 and 8.15.3 stack versions.

Mapping created for this field in this package during tests:

        "process": {
          "properties": {
            "pid": {
              "type": "float"
            },
            "title": {
              "type": "keyword",
              "fields": {
                "text": {
                  "type": "match_only_text"
                }
              }
            }
          }
        }

EDIT: It looks like that currently (with fields validation), this is accepted if the definition is long (ECS) and the value has type float:

https://github.com/elastic/elastic-package/blob/8cc126ae5015dd336b22901c365e8c98db4e7c15/internal/fields/validate.go#L1234-L1247

mrodm commented 1 week ago

Triggered a new execution in integrations with the rest of packages: https://buildkite.com/elastic/integrations/builds/18617

mrodm commented 4 days ago

In the latest integrations build, there are some failures in network_traffic package. These look related to the same issue https://github.com/elastic/elastic-package/pull/2214#issuecomment-2493731704

There are fields undefined under a group field, that I think thy are not taken into account since until spec 3.0.1 they are not checked.

Definition from ecs: https://github.com/elastic/ecs/blob/ce703ab38f0c128c0dfeeb2daf0893ff3021cdff/generated/ecs/ecs_flat.yml#L2204-L2222

EDIT: dns.answers in this case it's an array of objects: https://github.com/elastic/integrations/blob/ef28635e4ed1fd875b50861e7bd783af7ef68968/packages/network_traffic/data_stream/dns/sample_event.json#L27

Not sure how to detect this via the actual or preview mappings

elastic / elastic-package