Open bhavyarm opened 2 years ago
Pinging @elastic/kibana-operations (Team:Operations)
I'm pretty sure the problem is that there are so many saved objects to delete, and there isn't any bulk delete API. @elastic/kibana-core can we discuss the option of closing https://github.com/elastic/kibana/issues/30503 again?
can we discuss the option of closing https://github.com/elastic/kibana/issues/30503 again
I've added it to our sync.
I'm pretty sure the problem is that there are so many saved objects to delete, and there isn't any bulk delete API.
I tend to agree that this is the issue here. Deleting 10k objects without bulk APIs will likely take quite some time.
can we discuss the option of closing https://github.com/elastic/kibana/issues/30503 again?
I can close the issue for sure, but you probably meant to resolve it :trollface: ?
Realize this is from a few weeks ago... but could an alternative solution for this test be to use a space and delete it?
Does deleting a space clean up their saved objects in bulk? That sounds like a reasonable idea to me
@tylersmalley @spalger Yes, deleting a space will delete all its documents too.
One edge case is creating an object and then sharing it to another space. In this case, removing the original space won't delete the object. But outside of the spaces test suites I doubt this happens often.
FYI folks, https://github.com/elastic/kibana/pull/139680 merged in time for 8.5 FF. A few caveats with the API that are worth mentioning:
hidden
objects cannot be deleted using the API, these are filtered out from the permissible SO types we act onforce
. It's a similar option as to single delete except that the behavior applies to the whole request. If you want to be selective over which objects to force delete, you'll have to delete them one by one (or use the space-delete "hack")pMap
to minimize the number of concurrent requests. If anyone wants to help out and feels up to modifying the deleteByQuery
script, feel free to do so, otherwise, we'll try to get there as priorities allow.While discussing another issue, I think it was @pmuellr that mentioned that we really shouldn't delete alerts saved objects through the saved object API because it orphaned the task manager tasks related to it. So to really "clean" or "tear-down" after a test we should include things outside the .kibana index. Maybe we develop a function or service that takes all these things into account like;
For each space:
kibanaServer.savedObjects.cleanStandardList()
method currently doesWhile discussing another issue, I think it was @pmuellr that mentioned that we really shouldn't delete alerts saved objects through the saved object API because it orphaned the task manager tasks related to it.
correct. We also have a parallel set of APIs for connectors - they can be deleted as plain old SO's today, but that could change in the future, so probably connectors should be looped through and deleted through their APIs as well.
For each space:
- find any alerts and delete them using the proper API so that it cleans up any tasks related to them
- for other types of saved objects, use the proper APIs to remove them
- for any remaining saved objects, use the saved object API to delete them (like the
kibanaServer.savedObjects.cleanStandardList()
method currently does
Or, for each space, for each saved object, if it's a "special" type call it's special delete, otherwise call the SO delete. I believe we have or will have bulk delete for rules (maybe not connectors), and plain old SO's as well, so probably want to batch those up instead of deleting one at a time.
we have or will have bulk delete
@pmuellr there is a bulkDelete
saved objects API in main: https://github.com/elastic/kibana/pull/139680.
I also added docs for that: https://github.com/elastic/kibana/pull/141164
This bug report hasn't received any updates for a year. Can someone please verify if this is still relevant?
I am migrating test/api_integration/fixtures/es_archiver/management/saved_objects/scroll_count to kbnArchiver (kibana/test/api_integration/apis/saved_objects_management/scroll_count.ts) And this test is using await esArchiver.emptyKibanaIndex();.
I tried to replace it with await kibanaServer.savedObjects.cleanStandardList(); but the test failed. This is the test run with kibanaServer.savedObjects.cleanStandardList();
And this is the test run with await esArchiver.emptyKibanaIndex();
Creating an issue because we want to get rid of esArchiver.