[osquery] Migrate from custom telemetry sender to the new Core-provided one

afharo commented 2 years ago

Hi folks!

I'm creating this issue to track the effort for migrating the current custom telemetry sender in x-pack/plugins/osquery/server/lib/telemetry/sender.ts to the new Core APIs to send telemetry events: core.analytics.

The full documentation can be found at https://docs.elastic.dev/telemetry/collection/event-based-telemetry

Essentially, during the setup phase of your plugin, you need to register the event and the structure that you're planning to send via:

export class MyPlugin extends Plugin {
...
  public setup(core, pluginDependencies) {
    core.analytics.registerEventType({
      eventType: 'my_osquery_event_name',
      schema: {...}, // Declaration of the structure of the event
    });
  }
...
}

Then, anytime you need to ship any events, you can call core.analytics.reportEvent('my_osquery_event_name', {...contentOfTheEvent});. The API reportEvent is available in the setup and start contracts.

Some relevant documentation about the APIs can be found in the client's package readme: https://github.com/elastic/kibana/blob/main/packages/analytics/client/README.md#reporting-events:

Added benefits

You can send telemetry directly from the browser when the events are related to UI actions.
This API has the concept of Context Providers (refer to https://github.com/elastic/kibana/blob/main/packages/analytics/client/README.md#enriching-events for how to implement them). It allows enriching all the events with relevant data like the already existing license, cloud deployment ID, cluster_uuid, version, server's status, and many more, so you don't need to provide it in your own events.

Changes to your current implementation

Looking at your implementation, you would call events.forEach((event) => core.analytics.reportEvent(eventType, event)) inside your queueTelemetryEvents method, and you can delete all the rest 😇

Remember that, during the setup phase, you'll need to declare all the event types you report via this sender (FWIW, I couldn't find this sender used at all using my IDE code references 🤷).

Testing

There are some Jest mocks you can use for unit tests. They are exported alongside the core mocks.

But we also offer some FTR helpers to make developing FTR tests much easier. You can find them in test/analytics in the Kibana repo.

Indexer considerations

⚠️ You may need to create a new indexer for this new implementation considering these 2 changes:

The structure of the events with the new API looks like this:

{
  "event_type": "my_osquery_event_name",
  "timestamp": "2022-08-03T00:00:00.000Z", // <- automatically populated the moment you call `reportEvent`
  "context": {...}, // All the info coming from the Context Providers
  "properties": {...} // The event data provided as the second parameter in the call to `reportEvent`.
}

The target channels that the core API sends this data to are kibana-server and kibana-browser, depending on where the event was generated (on the server or the browser, respectively).

elasticmachine commented 2 years ago

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine commented 2 years ago

Pinging @elastic/security-asset-management (Team:Asset Management)

afharo commented 1 year ago

@patrykkopycinski do you think we can close this issue after https://github.com/elastic/kibana/pull/138221 was merged and backported?

elastic / kibana