Define and support a standard Elastic Agent input logging schema

Originally suggested by @andrewkroh in https://github.com/elastic/elastic-agent/issues/3640#issuecomment-1781875889

As a user performing troubleshooting, I would like to be able to correlate the stream ID from the agent policy with both the logs and metrics. IIRC from the last time I was trying to do this in Kibana, there were a few different fields that held this ID value and not all of them were indexed.

I think it would be helpful for Elastic Agent to give guidance on logging schema for its constituent inputs to follow. With that written down somewhere we could then make changes to ensure that it is followed and make sure those fields are mapped consistently in the elastic_agent integration. Here's a quick idea building upon on the component fields I see in the elastic_agent logs.

Field	Description	Examples
component.kind	Kind of component. These are general high-level concepts.	input, processor, output
component.type	Type of component. Type specifies a named implementation of the specific component kind.	winlog, filestream, add_cloud_metadata, elasticsearch
component.id	A unique identifier associated with an instance of of the component type.	winlog-windows.sysmon_operational-9c18c120-174e-11ee-925f-b5a3d2f00fd4
component.binary	Name of the binary without OS specific file extensions.	filebeat, auditbeat, elastic-defend

Example of data from beats:

[
  {
    "@timestamp": "2023-10-26 20:09:25 UTC",
    "component": {
      "id": "winlog-windows.sysmon_operational-9c18c120-174e-11ee-925f-b5a3d2f00fd4",
      "kind": "input",
      "type": "winlog"
    },
    "log": {
      "level": "info"
    },
    "message": "Hello world!"
  },
  {
    "@timestamp": "2023-10-26 20:09:25 UTC",
    "component": {
      "id": "winlog-windows.sysmon_operational-9c18c120-174e-11ee-925f-b5a3d2f00fd4",
      "kind": "input",
      "type": "winlog"
    },
    "event": {
      "kind": "metric"
    },
    "input": {
      "errors_total": 0,
      "received_events_total": 446799
    }
  },
  {
    "@timestamp": "2023-10-26 20:09:25 UTC",
    "component": {
      "id": "add_cloud_metadata-001",
      "kind": "processor",
      "type": "add_cloud_metadata"
    },
    "log": {
      "level": "debug"
    },
    "message": "Detected AWS"
  }
]

The component.x labels coming from the raw Beat logs would take precedence over any component.x labels being added by Elastic Agent as the logs flows through it.

elastic / elastic-agent

Define and support a standard Elastic Agent input logging schema #3768