[FR] Re-factor Bundled Schema Loading and Availability

[terrancedejesus]

Related

• https://github.com/elastic/detection-rules/issues/3222

Summary

For rule validation, several schemas (listed below) are loaded to do query field validation.

• ECS: Elastic Common Schema
• Beats: Beats Schema
• Non-ECS: Fields not defined in ECS
• Integrations: Integration defined schemas

The existing process has been matured since inception to support ECS, beats and Non-ECS where schemas are merged together and flattened, then made available throughout the repository for unit testing, field validation and more. Integration schemas are fairly new and thus have not been integrated resulting in redundant schema loading and validation in several areas of the code.

Goal: Expose a method to support loading all 4 schemas and make available where necessary. We should then review where this call can be used and adjust as necessary.

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[terrancedejesus]

This has been completed by @Mikaayenson. During rule validation, we now ensure that if an integration or data source is listed as compatible we check each schema and ensure compatibility of the query across each. Reference: https://github.com/elastic/detection-rules/pull/3372