search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.64k stars 8.23k forks source link

[Fleet] Prepare Fleet API for promotion to GA #123150

Closed joshdover closed 1 year ago

joshdover commented 2 years ago

One design goal of Fleet is to be 100% API-based in our business logic. This allows for easy external integration and management of the Fleet platform and paves the way for future 1st class features, such as Infrastructure-as-Code (IaC) tooling support (Terraform, etc.).

Today, Fleet publishes an OpenAPI spec which is currently marked as experimental.

There will be a beta period prior to the Fleet API's general availability. This beta period is intended to provide internal and external consumers of the Fleet API some time to begin adopting new features, and to provide feedback and input on the transition to General Availability.

:one: Required for beta

Beta requirements are now fulfilled. These issues _must_ be completed before the Fleet API can enter a formal beta period. ### "Recipe" documentation We need some level of "how to use this API" documentation around the Fleet API that captures common operations and processes. These docs should include all of the following operations: - Add new integration - Update an Integration Config - Upgrade installed integration - Delete an integration config - Delete installed integrations - Create new agent policy - Update agent policy - Reassign agent - Update agent config - Upgrade agent version - Unenroll agent - Delete agent policy - Delete agent ### Package Policy API improvements and breaking changes The package policy API is difficult to use programmatically. It requires too much knowledge of the structure of specific integrations and their inputs. It should improved top-to-bottom to better support automation use cases at scale. Several of the changes below will be considered breaking, so we need to handle these before we can consider the Fleet API ready for a beta period. - [x] https://github.com/elastic/kibana/issues/132263 - [x] https://github.com/elastic/kibana/issues/136804 - [x] https://github.com/elastic/kibana/issues/136945 - [x] https://github.com/elastic/kibana/issues/137005 ### Audit OpenAPI spec and remove unused endpoints Any endpoint that is completely unused or doesn't have a use-case should be removed prior to beta to limit the API's surface area. ### Update bulk actions API In supporting https://github.com/elastic/kibana/issues/141567, we'll likely be making breaking changes to the bulk actions API response structure. We should make sure these changes land and documentation is accurate before we consider the Fleet API beta. cc @juliaElastic.

:two: Required for GA

The GA requirements are now fulfilled These issues don't block the initial beta, but they should be completed prior to the Fleet API transition into General Availability. ### First-party reference documentation Currently, Fleet maintains its own set of [OpenAPI](https://www.openapis.org/) documentation in GitHub [here](https://github.com/elastic/kibana/tree/main/x-pack/plugins/fleet/common/openapi). This documentation is not hosted or bundled with any Elastic-owned documentation. Typically, we use a [link](https://petstore.swagger.io/?url=https://raw.githubusercontent.com/elastic/kibana/main/x-pack/plugins/fleet/common/openapi/bundled.json#/) to a Swagger UI provider to allow for exploring the API spec. This is not scalable, and we need to provide first-party documentation for the Fleet API in order to promote it to GA. It'd be great to get API docs on https://docs.elastic.co, which supports MDX docs. It's not clear, though, how we'd integrate our OpenAPI spec files into this docs system currently. https://github.com/elastic/docs.elastic.co/issues/37 tracks the implementation for a React component that consumes OpenAPI specs and produces a UI within the docs.elastic.co site. - [x] https://github.com/elastic/kibana/issues/135519 - [x] https://github.com/elastic/kibana/issues/136807 - [x] https://github.com/elastic/kibana/issues/143579 Publish official documentation - We should be able to use the new documentation system's support for OpenAPI specs for this ### [No longer required] API versioning + breaking change management > **Note** > The decision making process for API versioning for Kibana API's is still ongoing as we start to plan for the Elastic Serverless offering. The Fleet API's promotion to GA should not be blocked by this effort. Another concern with the Fleet API moving forward is limiting breaking changes. There are two distinct efforts related to this: 1. Implementation of versioning within the Kibana API - this is a higher level effort spanning all Kibana plugins that expose an API 2. Automation and patterns around preventing/mitigating breaking changes from landing in the API - [ ] Determine a policy for deprecations and breaking changes that is compatible with Elastic's versioning policy for 8.0+ - [ ] Depending on outcome of above, add a versioning scheme (see prior art from Kibana Platform) - [ ] Prevent the possibility of unplanned breaking changes by adding CI checks for detecting breaking changes on the OpenAPI spec ### Other API fixes - [x] https://github.com/elastic/kibana/issues/113750 - [x] https://github.com/elastic/kibana/issues/119182 - [x] https://github.com/elastic/kibana/issues/136801 - [x] https://github.com/elastic/kibana/issues/136805 - [x] https://github.com/elastic/kibana/issues/135980 - [x] ~~Any breaking changes needed for RBAC evolution~~ (none anticipated)

Optional

These are "nice to have" improvements that aren't necessarily required for promoting the Fleet API to beta or GA.

juliaElastic commented 2 years ago

121485 has a few downstream dependencies, does Fleet API GA have to depend on it?

joshdover commented 2 years ago

Any breaking changes needed for RBAC evolution

@juliaElastic I don't believe we identified any needs for breaking changes on the HTTP API to support our RBAC evolution goals. Can you confirm and if so, check this one off the list?

juliaElastic commented 2 years ago

Any breaking changes needed for RBAC evolution

@juliaElastic I don't believe we identified any needs for breaking changes on the HTTP API to support our RBAC evolution goals. Can you confirm and if so, check this one off the list?

Yes, we don't plan any breaking changes for RBAC. I'll close that one out.

jen-huang commented 2 years ago

@juliaElastic @kpollich Would it make sense to also include these?:

jen-huang commented 2 years ago

@kpollich We've had a couple of recent issues related to the data stream API, we've discussed putting some time here to make it more fault-tolerant. Could improvements here be a candidate? Related issues:

juliaElastic commented 2 years ago

@juliaElastic @kpollich Would it make sense to also include these?:

I think the Optimize issue does not change the API itself, so it can be done independently of moving to GA. The deprecate issue can be included in this.

juliaElastic commented 2 years ago

@kpollich I added a few more enhancements to this issue:

joshdover commented 2 years ago

We may want to consider this one as well as a fix required for GA: https://github.com/elastic/kibana/issues/137005

kpollich commented 2 years ago

@joshdover Thanks - added to the description.

kpollich commented 2 years ago

It'd be great to integrate Fleet's OpenAPI docs with a first party Elastic docs site, preferably https://docs.elastic.co/. OpenAPI support isn't quite ready yet, but is planned and tracked here https://github.com/elastic/docs.elastic.co/issues/37.

We should be doing everything we can do clean up and improve our existing OpenAPI files to make sure they're ready for integration with a first-party docs site when possible.

jlind23 commented 2 years ago

@nimarezainia can we make sure that the docs are good to be shipped before 8.5 official release?

jlind23 commented 2 years ago

Update: Some more docs details will be added to: https://github.com/elastic/observability-docs/issues/2180 then our writers will take it from there and create some getting started materials.

As the steps needed to get it to a Beta stage are now all done, i'll move this issue to a later iteration.

cc @kpollich @jen-huang

kpollich commented 2 years ago

We need to make sure once we formally enter a beta period that any public-facing docs for the Fleet API or its OpenAPI spec to say beta rather than experimental.

nimarezainia commented 2 years ago

marking it beta is the easy part. What do we want to collect from customers? a sample workflow or test guide would be great.

kpollich commented 1 year ago

Relevant docs/OpenAPI issue: https://github.com/elastic/kibana/issues/137240

kpollich commented 1 year ago

We need to address some issues with Fleet's OpenAPI spec to unblock the docs team: https://github.com/elastic/kibana/issues/149990

jlind23 commented 1 year ago

@kpollich converting this to a meta issue as we already added the follow up ones in our sprint.

jen-huang commented 1 year ago

@kpollich Per discussion, going to close this issue for planning purposes. Feel free to reference any follow up docs issues needed.