Make sure that main ECS fields are mapped in the Elastic Agent integration

[rdner]

Recently we had a report that log.file.path is not mapped in events created in the Elastic-Agent integration (dataset: elastic_agent.endpoint_security-external).

Another report that error.message is not mapped in logs-elastic_agent.filebeat-syslog either.

All the logs-*-* data streams including logs-elastic_agent.endpoint_security-external are created from this logs-template.json

which is composed of these component templates:

• "logs-mappings"
• "logs-settings"
• "logs@custom" (ignored when missing)
• "ecs@dynamic_templates"

The last 2 were added ~4 months ago in this PR (should have been released with 8.9.x) https://github.com/elastic/elasticsearch/pull/96171

Which means the log.file.path field should be mapped via this dynamic mapping:

https://github.com/elastic/elasticsearch/blob/48fcb2e7d4651b95a8e22612e2ec001ad4c4bdc4/x-pack/plugin/core/template-resources/src/main/resources/ecs-dynamic-mappings.json#L100

However, I see that dynamic mappings are disabled in the integration:

https://github.com/elastic/integrations/blob/ffe9c35004696815c24fdb89aa86778d2ea67885/packages/elastic_agent/data_stream/endpoint_sercurity_logs/manifest.yml#L7

It seems we have two options (both need to be tested):

1. Set dynamic to true here https://github.com/elastic/integrations/blob/ffe9c35004696815c24fdb89aa86778d2ea67885/packages/elastic_agent/data_stream/endpoint_sercurity_logs/manifest.yml#L7
2. Add a limited list of most important fields (e.g. log.file.path) here https://github.com/elastic/integrations/blob/ffe9c35004696815c24fdb89aa86778d2ea67885/packages/elastic_agent/data_stream/endpoint_sercurity_logs/fields/ecs.yml

(1) might have big impact and lead to performance issues, see https://github.com/elastic/integrations/pull/1298#issuecomment-881096902

(2) is fixing the currently reported symptom instead of the root cause and we might receive more reports in the future.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

I vote for option 2 unless we believe option 1 is totally risk free. We don't seem to get a lot of requests for new fields, so I hesitate to do something that may cause performance problems whenever the elastic_agent integration is used (which is a lot).

[rdner]

We've got another report that error.message is not mapped either.

Mapped only in the fleet server logs https://github.com/elastic/integrations/blob/399ac66a2c9624c17b66cca343110388e7ec23e2/packages/elastic_agent/data_stream/fleet_server_logs/fields/ecs.yml#L17