[Change Proposal] Support ML trained models in integration packages

[alvarezmelissa87]

Please read the section on Change Proposals in the Contributing Guide and flesh out this issue accordingly. Thank you!

What problem the proposal is solving.

Users should be able to easily install supervised ML models - ML Domain Generated Algorithm detection model and Problem child - and related assets. This requires creating packages for each of these supervised models. Integrations will need to support the installation of these models.

Where the solution will need to be implemented, i.e. which parts, if any, of the Elastic Stack will be impacted.

• [x] Create PR in the https://github.com/elastic/package-spec repo to define the ml_model asset type

  • [x] PR - https://github.com/elastic/package-spec/pull/204

• [x] Create PR in https://github.com/elastic/kibana to add support (installing the asset using the existing create trained model API) for the model asset type into Fleet

  • [x] PR - https://github.com/elastic/kibana/pull/107710

• [ ] Create PR in https://github.com/elastic/integrations to add the model JSON containing the model and other model package assets themselves

• After testing the module, promote to the package to the package registry service for first the snapshot environment, then staging, and finally the production environment.

[alvarezmelissa87]

cc @joshdover, @jen-huang for Fleet plugin change proposal cc @mtojek for package spec changes

[mtojek]

I agree with @ruflin's comment on the execution order - let's decide on the model lifecycle in Kibana first. Approving the spec PR is formal here.

[ruflin]

It's great to see we already have a PR for Kibana on how installation etc. should be done: https://github.com/elastic/kibana/pull/107710 @alvarezmelissa87 Any chance you could quickly elaborate here without code on how installing, upgrade, removal is working and if there are any "special cases" also related to the ingest pipeline? What is the order etc?

[susan-shu-c]

@alvarezmelissa87 this can be closed right? I'm curious as well if this only supports the dataframe analytics, and does it support PyTorch models like how we can do with command line or eland? link