[Security Solution][Telemetry] Add integration tests

szaffarano commented 1 week ago

Summary

Improve the telemetry logging by including execution context as structured arguments to facilitate searching by those fields. The TelemetryLogger now includes always by default some basic data (like cluster uuid), so it is highly recommended to start using it when possible (instantiate it using the newTelemetryLogger function)
Start using the above feature in the receiver as well as in some tasks.
Add integration tests mainly for endpoint-meta-telemetry
Refactor endpoint-meta-telemetry to simplify its logic and also improve error handling. Based on the tests, I had to add or change how we managed errors for some (edge) cases. Also, the errors now log more information introducing fields for relevant data instead of using a nonstatic string (e.g., instead of Running task: ${taskId} [last: ${taskExecutionPeriod.last} - current: ${taskExecutionPeriod.current}], now taskId and the executionPeriod are searchable fields and not part of the log message, which becomes a "static string"). Regarding error handling, the goal is to send as much information as possible and not abort the execution when we encounter weird (but still feasible) errors.
Enable the async sender in endpoint-meta-telemetry using the already defined feature flag, just in case we need to disable it through a configuration artifact. This change deprecates the current way we split the events to send them into batches (although if we disable the feature flag, it still uses this approach).
Integration tests:

elasticmachine commented 1 week ago

Pinging @elastic/security-solution (Team: SecuritySolution)

kibana-ci commented 1 week ago

:green_heart: Build Succeeded

Buildkite Build
Commit: b99156bf1076af58a06becba1e092bcefffde95a

Metrics [docs]

Unknown metric groups

#### ESLint disabled line counts | id | [before](https://github.com/elastic/kibana/commit/70fb22dfd8f09d601fa0ce7c9db35e90f79efdcb) | [after](https://github.com/elastic/kibana/commit/b99156bf1076af58a06becba1e092bcefffde95a) | diff | | --- | --- | --- | --- | | `securitySolution` | 517 | 518 | +1 | #### Total ESLint disabled count | id | [before](https://github.com/elastic/kibana/commit/70fb22dfd8f09d601fa0ce7c9db35e90f79efdcb) | [after](https://github.com/elastic/kibana/commit/b99156bf1076af58a06becba1e092bcefffde95a) | diff | | --- | --- | --- | --- | | `securitySolution` | 595 | 596 | +1 |