search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.73k stars 8.14k forks source link

[Fleet] [Meta] Fleet API Integration Tests #151178

Open kpollich opened 1 year ago

kpollich commented 1 year ago

Fleet currently maintains a large suite of API integration tests using Kibana's "functional test runner" (FTR). With the approach of the Fleet API's transition to general availability in #123150, we'll want to make sure our test coverage around all Fleet API endpoints is solid.

Helpful links

The following is an initial audit of the existing integration tests for each API resource. There may be cases where we want to improve or extend existing tests, even if they're checked off here. The test suites are ordered roughly around their overall level of importance/priority as they relate to the overall Fleet product.

## Agent APIs
- [x] `GET /agents` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/list.ts#L20)
- [x] `POST /agents/bulk_upgrade` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/upgrade.ts#L349)
- [x] `GET /agents/action_status` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/action_status.ts#L32)
- [ ] `GET /agents/:id`
- [x] `PUT /agents/:id` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/update.ts#L16)
- [x] `DELETE /agents:id` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/delete.ts#L41)
- [x] `POST /agents/:id/actions` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/actions.ts#L28)
- [x] `POST /agents/:agentId/actions/:actionId/cancel` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/actions.ts#L105)
- [x] `GET /agents/files/:fileId/:fileName` - [Link](https://github.com/elastic/kibana/blob/fdd709b02579b05124585b10fa6535d0a90480e2/x-pack/test/fleet_api_integration/apis/agents/uploads.ts#L93)
- [x] `PUT /agents/:id/reassign` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/reassign.ts#L37)
- [x] `POST /agents/:id/unenroll` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/unenroll.ts#L21)
- [x] `POST /agents/:id/upgrade` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/upgrade.ts#L59)
- [x] `GET /agents/:id/uploads` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/uploads.ts#L26)
- [x] `POST /agents/bulk_reassign` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/reassign.ts#L91)
- [x] `POST /agents/bulk_unenroll` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/unenroll.ts#L114-L226) (needs refactor to separate `describe` blocks)
- [X] `POST /agents/bulk_update_agent_tags` - [Link](https://github.com/elastic/kibana/blob/fdd709b02579b05124585b10fa6535d0a90480e2/x-pack/test/fleet_api_integration/apis/agents/update_agent_tags.ts#L19)
- [x] `GET /agents/tags` - [Link](https://github.com/elastic/kibana/blob/fdd709b02579b05124585b10fa6535d0a90480e2/x-pack/test/fleet_api_integration/apis/agents/list.ts#L127-L130)
- [x] `POST /agents/:id/request_diagnostics` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/request_diagnostics.ts)
- [x] `POST /agents/bulk_request_diagnostics` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/request_diagnostics.ts) (needs refactor to separate `describe` blocks)
- [ ] `GET /agents/setup` - Maybe not needed?
- [x] `POST /agents/setup` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/fleet_setup.ts)
- [x] `GET /agent_status` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agents/status.ts)
- [ ] `GET /agent_status/data`
## Agent Policy APIs
- [ ] `GET /agent_policies`
- [ ] `GET /agent_policies/:id`
- [x] `PUT /agent_policies/:id` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agent_policy/agent_policy.ts#L634)
- [x] `POST /agent_policies` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agent_policy/agent_policy.ts#L24)
- [x] `POST /agent_policies/copy` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agent_policy/agent_policy.ts#L347)
- [ ] `GET /agent_policies/full`
- [ ] `GET /agent_policies/download`
- [x] `POST /agent_policies/_bulk_get` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agent_policy/agent_policy.ts#L922)
- [x] `POST /agent_policies/delete` - [Link](https://github.com/elastic/kibana/blob/587816830afdb70f66ed058bb009f942c0bab3ea/x-pack/test/fleet_api_integration/apis/agent_policy/agent_policy.ts#L866)
## Package Policy APIs
- [ ] `GET /package_policies`
- [x] `POST /package_policies` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/package_policy/create.ts#L26)
- [x] `POST /package_policies/_bulk_get` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/package_policy/get.ts#L156)
- [x] `POST /package_policies/upgrade` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/package_policy/upgrade.ts#L164)
- [x] `POST /package_policies/upgrade/dryrun` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/package_policy/upgrade.ts#L140)
- [x] `GET /package_policies/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/package_policy/get.ts#L39)
- [x] `PUT /package_policies/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/package_policy/update.ts#L56)
- [x] `DELETE /package_policies/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/package_policy/delete.ts#L19)
- [x] `POST /package_policies/delete` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/package_policy/delete.ts#L152)
## Enrollment API Key APIs
- [x] `GET /enrollment_api_keys` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/enrollment_api_keys/crud.ts#L37)
- [x] `POST /enrollment_api_keys` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/enrollment_api_keys/crud.ts#L123)
- [x] `GET /enrollment_api_keys/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/enrollment_api_keys/crud.ts#L63)
- [x] `DELETE /enrollment_api_keys/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/enrollment_api_keys/crud.ts#L87)
## EPM APIs
- [ ] `GET /epm/categories`
- [ ] `GET /epm/packages/limited` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/epm/list.ts#L48) (needs more coverage)
- [x] `POST /epm/packages` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/epm/install_by_upload.ts#L67)
- [x] `POST /epm/packages/_bulk` - [Link](https://github.com/elastic/kibana/blob/main/x-pack/test/fleet_api_integration/apis/epm/bulk_upgrade.ts)
- [x] `GET /epm/packages/:name/:version` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/epm/get.ts#L42)
- [x] `POST /epm/packages/:name/:version` - See `install_` files in [EPM tests directory](https://github.com/elastic/kibana/tree/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/epm) for cases covered
- [ ] `PUT /epm/packages/:name/:version`
- [x] `DELETE /epm/packages/:name/:version` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/epm/delete.ts#L34)
- [ ] `GET /epm/packages/:name/:version/:filePath`
- [ ] `GET /epm/packages/:name/status`
## Output APIs
- [x] `GET /outputs` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/outputs/crud.ts#L45)
- [x] `POST /outputs` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/outputs/crud.ts#L150)
- [x] `GET /outputs/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/outputs/crud.ts#L53)
- [x] `DELETE /outputs/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/outputs/crud.ts#L410)
- [x] `PUT /outputs/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/outputs/crud.ts#L63)
## Agent download source APIs
- [x] `GET /agent_download_sources` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/download_sources/crud.ts#L46)
- [x] `POST /agent_download_sources` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/download_sources/crud.ts#L138)
- [x] `GET /agent_download_sources/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/download_sources/crud.ts#L61)
- [x] `DELETE /agent_download_sources/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/download_sources/crud.ts#L203)
- [x] `PUT /agent_download_sources/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/download_sources/crud.ts#L78)
## Fleet server host APIs
- [x] `GET /fleet_server_hosts` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/fleet_server_hosts/crud.ts#L61)
- [ ] `POST /fleet_server_hosts`
- [x] `GET /fleet_server_hosts/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/fleet_server_hosts/crud.ts#L69)
- [ ] `DELETE /fleet_server_hosts/:id`
- [x] `PUT /fleet_server_hosts/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/fleet_server_hosts/crud.ts#L91)
## Proxy APIs
- [x] `GET /proxies` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/fleet_proxies/crud.ts#L103)
- [ ] `POST /proxies`
- [x] `GET /proxies/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/fleet_proxies/crud.ts#L111)
- [x] `DELETE /proxies/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/fleet_proxies/crud.ts#L167)
- [x] `PUT /proxies/:id` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/fleet_proxies/crud.ts#L132)
## Settings APIs
- [x] `GET /settings` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/settings/get.ts#L19)
- [x] `PUT /settings` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/settings/update.ts#L22) (Tests are skipped, should be removed)
## Misc APIs
- [x] `GET /data_streams` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/data_streams/list.ts#L108)
- [ ] `POST /setup` - Only 200 response is tested
- [ ] `POST /health_check`
- [x] `POST /service_tokens` - [Link](https://github.com/elastic/kibana/blob/main/x-pack/test/fleet_api_integration/apis/service_tokens.ts)
- [x] `POST /logstash_api_keys` - [Link](https://github.com/elastic/kibana/blob/11dc7895374307303627b9bbba47fc988d58dc4d/x-pack/test/fleet_api_integration/apis/outputs/logstash_api_keys.ts)
- [ ] `GET /kubernetes` 
## Other improvements
- [ ] Ensure all test suite names use the format `{VERB} /api/fleet/{...endpoint}`, and ensure Fleet's test output is meaningful + consistent
elasticmachine commented 1 year ago

Pinging @elastic/fleet (Team:Fleet)

jlind23 commented 1 year ago

@juliaElastic now that Fleet API are GA, could we close this issue or are we still missing some important tests?

juliaElastic commented 1 year ago

@jlind23 I think we should review this list again to see which tests are missing.

criamico commented 1 year ago

@juliaElastic @jlind23 I went through this list as part of the work to determine the smoke tests needed for serverless. The empty checkboxes are an accurate representation of missing integration tests.

jlind23 commented 1 year ago

@criamico @juliaElastic thanks for the feedback, let's try to spend time on this during the next following sprints.