search

elastic / package-spec

EPR package specifications
Other
18 stars 73 forks source link

Entity definition use case #783

Open smith opened 3 months ago

smith commented 3 months ago

Currently the Elastic Entity Model has hard-coded entity definitions (https://github.com/elastic/kibana/blob/main/x-pack/plugins/observability_solution/entity_manager/docs/entity_definitions.md)

We would like to have packages include entity definitions (for instance, a kubernetes package could contain entity definitions for k8s.pod, k8s.node, etc.)

While everything related to entities currently happens in Kibana/ES, it would also be possible to do entity discovery in the otel collector, in which case we would want to distribute the entity definition assets to the collectors.

jsoriano commented 3 months ago

How do entity definitions look like? Is it enough with a JSON like the one linked below, or JS code is also needed? https://github.com/elastic/kibana/blob/a8aa215db5b26b1dcdb049231dad2ae787167048/x-pack/plugins/observability_solution/entity_manager/server/lib/entities/built_in/services.ts#L22

smith commented 3 months ago

@jsoriano that's the right place. It's just JSON, and a REST API.

There's an open PR with the OpenAPI docs: https://github.com/elastic/kibana/pull/190203 that's probably the best place to look now. You can clone that branch and look at the docs locally or just read the JSDoc comments in the TS server route definitions.

So the UI for defining entities at this point is creating json files and posting them to a Kibana API.

I'm not sure where docs for the entities schema are, but you can probably figure it out from https://github.com/elastic/kibana/blob/main/x-pack/packages/kbn-entities-schema/src/schema/entity_definition.ts.

klacabane commented 2 months ago

As far as implementation goes, we should be able to copy the SLO implementation with a few minor tweaks since they share the same use case. This means supporting creation of a <package>/kibana/entity_definition folder with json files

klacabane commented 2 months ago

@jsoriano since entity definitions are tech preview atm and we expect a few (breaking) changes to be introduced fairly quickly, is there a way to specify that in the package-spec (ie anything under kibana/entity_definition folder is not production ready) ? Something similar to the release property that one can add to a package root configuration, or a data stream ?

Also if I look at the SLO package-spec addition, SLOs are added as saved object format - is that a requirement or just a convenient way to handle the blob in fleet (since everything else under kibana/.. is a saved object) ? Is there anything preventing us from dumping a custom json schema as long as we handle it correctly in fleet ?

jsoriano commented 2 months ago

is there a way to specify that in the package-spec (ie anything under kibana/entity_definition folder is not production ready) ? Something similar to the release property that one can add to a package root configuration, or a data stream ?

Yes, there are two possible approaches, one is to use release: beta in the spec of the directory, as we do with content packages in https://github.com/elastic/package-spec/blob/d9888bd70ae1d344a98ffc1dae08eccc1f90ad63/spec/content/spec.yml#L14 The other one is to add the feature to a future version, as the features we have reserved for 3.3 and 3.4: https://github.com/elastic/package-spec/blob/d9888bd70ae1d344a98ffc1dae08eccc1f90ad63/spec/changelog.yml#L5 These are controlled with JSON patches like these ones: https://github.com/elastic/package-spec/blob/d9888bd70ae1d344a98ffc1dae08eccc1f90ad63/spec/integration/kibana/spec.yml#L140

These declarations prevent the publication of GA packages that use these features, but don't prevent the features to be used in prerelease packages.

Also if I look at the SLO package-spec addition, SLOs are added as saved object format - is that a requirement or just a convenient way to handle the blob in fleet (since everything else under kibana/.. is a saved object) ? Is there anything preventing us from dumping a custom json schema as long as we handle it correctly in fleet ?

This was convenient for the SLO case but is not a requirement, there is nothing preventing other approaches. In any case changes in Kibana might be needed.