Open kibanamachine opened 8 months ago
Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)
From the 4 instances above, 2 of them were wrong values in test that was fixed in the PR, and for the other 2, the root cause mentioned in the logs is:
index_not_found_exception: no such index [alert-action-threshold]
In both cases, we have the following error log in task manager:
[00:02:02] │ proc [kibana] [2024-01-29T10:53:05.557+00:00][ERROR][plugins.taskManager] Task actions:.index "7484ee48-6ac2-4c5c-a778-8fef443c4025"
failed: Error: Saved object [action/6a53161a-9e45-450e-b77a-bc3bfa399b96] not found
{"tags":["actions:.index","7484ee48-6ac2-4c5c-a778-8fef443c4025","task-run-failed"],
"error":{"stack_trace":"Error: Saved object [action/6a53161a-9e45-450e-b77a-bc3bfa399b96] not found\n
at Function.createGenericNotFoundError (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/core-saved-objects-server/src/saved_objects_error_helpers.js:236:37)\n at performGet (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/core-saved-objects-api-server-internal/src/lib/apis/get.js:73:60)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at SavedObjectsRepository.get (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/core-saved-objects-api-server-internal/src/lib/repository.js:218:12)\n at Object.getDecryptedAsInternalUser (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/encrypted-saved-objects-plugin/server/saved_objects/index.js:50:29)\n at ActionExecutor.getActionInfoInternal (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/actions-plugin/server/lib/action_executor.js:377:25)\n at Object.loadIndirectParams (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/actions-plugin/server/lib/task_runner_factory.js:81:30)\n at TaskManagerRunner.validateIndirectTaskParams (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:419:11)\n at TaskManagerRunner.run (/var/lib/buildkite-agent/builds/kb-n2-4-spot-17cd2aa3ed32e2f7/elastic/kibana-on-merge/kibana-build-xpack/node_modules/@kbn/task-manager-plugin/server/task_running/task_runner.js:320:34)"},"service":{"node":{"roles":["background_tasks","ui"]}}}
Started a discussion in Slack.
It seems eventually a document is found but the information in that document is not as we expetected:
debg Found 1 docs, looking for atleast 1.
I made some improvements in this PR and will close this issue as I didn't see a failure when I ran the test 200 times after the improvement. I've also added some logs, so if it happens again, hopefully, it will be easier to investigate it.
New failure: CI Build - 8.14
A test failed on a tracked branch
First failure: CI Build - main