Agent health metrics

[felixbarny]

Define a standard set of metrics that each agent should eventually send

• Metrics about event handling, queuing, and sending to APM Server
• Agent CPU usage (stretch)

Creating a custom dashboard that shows the metrics and publish it to elastic/apm-contrib:

• [x] https://github.com/elastic/apm/issues/633

Spec issue

• [x] #626

Agent issues

• [ ] https://github.com/elastic/apm-agent-java/issues/2543

[SylvainJuge]

Few ideas that might be relevant to add:

• number of bytes send/received by the agent: knowing how much data is sent/received on the network by agent seems relevant (less if there is a local apm server, but still a good proxy on the amount of data we capture).
• number of network errors when sending data or fetching configuration, while the APM server might be overloaded or not accessible it it's at least able to partially capture parts of those it could help with intermittent network issues or load issues with APM server.
• number of traced/captured spans and transactions: help to know if everything seen by the agent has been sent to the server, might also be relevant to help with high-level integration tests where the numbers should match or how the sampling is being applied.

Would very likely overlap/replace the previous issue on the topic : https://github.com/elastic/apm/issues/160