[Change Proposal] Validate top-level keys in Ingest Pipeline definition

[andrewkroh]

Currently there is no validation of the keys contained in Ingest Pipeline definitions. I propose we validate the top-level pipeline keys that we want to allow (description, processors, on_failure). We don't need validate deeper into the processors at this time.

Elasticsearch supports more key(s) like version, and I have seen this present in some pipeline definitions. But I think this version key should be prohibited in Fleet packages. It does not appear to be of use to Fleet and is confusing to have a version embedded in the pipeline that differs from the package version.

I discovered the version key already being used in a few places:

• https://github.com/elastic/integrations/blob/f6088b98361e419a3750a789ad84fdefb13c43eb/packages/bitdefender/data_stream/push_configuration/elasticsearch/ingest_pipeline/default.yml#L2
• https://github.com/elastic/integrations/blob/c3b6d03b3d6a40baf86c866e5ff989e08613b164/packages/forcepoint_web/data_stream/logs/elasticsearch/ingest_pipeline/default.yml#L2
• https://github.com/elastic/integrations/blame/c3b6d03b3d6a40baf86c866e5ff989e08613b164/packages/jumpcloud/data_stream/events/elasticsearch/ingest_pipeline/default.yml#L2
• https://github.com/elastic/integrations/blob/e6b12b116955ad8539858fb78989be47fead3fd0/packages/tines/data_stream/audit_logs/elasticsearch/ingest_pipeline/default.yml#L2

[andrewkroh]

Implemented by https://github.com/elastic/package-spec/pull/586