field collisions with organization.id & elastic maintained integrations

[neu5ron]

organization.id is a common field that is used in multi-tenant environments. However, there are multiple elastic managed pipelines (o365 audit, cisco meraki, google workspace, zscaler, and possibly more) that try to set organization.id and fails if the organization.id already exists. It should be noted in the ECS documentation to not set this field upfront or pipelines should be changed. I opened an issue in ECS repo as well https://github.com/elastic/ecs/issues/2250 as not sure which party should be involved.

Here are referenced lines: https://github.com/elastic/integrations/blob/b50c74066d3cca005259bcfccd7543b9dc4a107b/packages/cisco_meraki/data_stream/events/elasticsearch/ingest_pipeline/default.yml#L35

https://github.com/elastic/integrations/blob/b50c74066d3cca005259bcfccd7543b9dc4a107b/packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml#L73

https://github.com/elastic/integrations/blob/b50c74066d3cca005259bcfccd7543b9dc4a107b/packages/google_workspace/data_stream/admin/elasticsearch/ingest_pipeline/default.yml#L71

https://github.com/elastic/integrations/blob/b50c74066d3cca005259bcfccd7543b9dc4a107b/packages/google_workspace/data_stream/alert/elasticsearch/ingest_pipeline/default.yml#L115

https://github.com/elastic/integrations/blob/b50c74066d3cca005259bcfccd7543b9dc4a107b/packages/google_workspace/data_stream/user_accounts/elasticsearch/ingest_pipeline/default.yml#L83

and every other google_workspace besides default.yml

zscaler audit actually seems to have proper logic that could be implemented across the board for the others (I would assume, but you all know best): https://github.com/elastic/integrations/blob/b50c74066d3cca005259bcfccd7543b9dc4a107b/packages/zscaler_zpa/data_stream/audit/elasticsearch/ingest_pipeline/default.yml#L74

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)