Failing test: Jest Integration Tests.x-pack/plugins/fleet/server/integration_tests - Fleet preconfiguration reset Reset all policy Works and reset all preconfigured policies

kibanamachine commented 2 years ago

A test failed on a tracked branch

Error: Unable to setup docker registry
    at startDockerRegistryServer (/opt/local-ssd/buildkite/builds/kb-n2-4-0a27290433d2da79/elastic/kibana-hourly/kibana/x-pack/plugins/fleet/server/integration_tests/docker_registry_helper.ts:50:11)
    at /opt/local-ssd/buildkite/builds/kb-n2-4-0a27290433d2da79/elastic/kibana-hourly/kibana/x-pack/plugins/fleet/server/integration_tests/docker_registry_helper.ts:61:5
    at _callCircusHook (/opt/local-ssd/buildkite/builds/kb-n2-4-0a27290433d2da79/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/run.js:175:5)
    at _runTestsForDescribeBlock (/opt/local-ssd/buildkite/builds/kb-n2-4-0a27290433d2da79/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/run.js:45:5)
    at _runTestsForDescribeBlock (/opt/local-ssd/buildkite/builds/kb-n2-4-0a27290433d2da79/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/run.js:57:9)
    at run (/opt/local-ssd/buildkite/builds/kb-n2-4-0a27290433d2da79/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/opt/local-ssd/buildkite/builds/kb-n2-4-0a27290433d2da79/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:176:21)
    at jestAdapter (/opt/local-ssd/buildkite/builds/kb-n2-4-0a27290433d2da79/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:109:19)
    at runTestInternal (/opt/local-ssd/buildkite/builds/kb-n2-4-0a27290433d2da79/elastic/kibana-hourly/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/opt/local-ssd/buildkite/builds/kb-n2-4-0a27290433d2da79/elastic/kibana-hourly/kibana/node_modules/jest-runner/build/runTest.js:472:34)

First failure: CI Build - main

elasticmachine commented 2 years ago

Pinging @elastic/fleet (Team:Fleet)

kibanamachine commented 2 years ago

New failure: CI Build - 8.1

jen-huang commented 2 years ago

See https://github.com/elastic/kibana/pull/125530.

kibanamachine commented 2 years ago

New failure: CI Build - 8.2

spalger commented 2 years ago

This, https://github.com/elastic/kibana/issues/124780, https://github.com/elastic/kibana/issues/134529, and https://github.com/elastic/kibana/issues/124781 all seem to be caused because the docker registry still fails to start for some reason:

{
  "ecs": {
    "version": "8.0.0"
  },
  "@timestamp": "2022-06-15T17:15:41.721-04:00",
  "message": "[Elastic Cloud agent policy 0001] could not be added. [fleet_server] could not be installed due to error: [PackageNotFoundError: [fleet_server] package not found in registry]",
  "error": {
    "message": "[Elastic Cloud agent policy 0001] could not be added. [fleet_server] could not be installed due to error: [PackageNotFoundError: [fleet_server] package not found in registry]",
    "type": "Error",
    "stack_trace": "Error: [Elastic Cloud agent policy 0001] could not be added. [fleet_server] could not be installed due to error: [PackageNotFoundError: [fleet_server] package not found in registry]\n    at /var/lib/buildkite-agent/builds/kb-n2-4-spot-ec638bb99f0408cd/elastic/kibana-on-merge/kibana/x-pack/plugins/fleet/server/services/preconfiguration.ts:212:21\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 0)\n    at ensurePreconfiguredPackagesAndPolicies (/var/lib/buildkite-agent/builds/kb-n2-4-spot-ec638bb99f0408cd/elastic/kibana-on-merge/kibana/x-pack/plugins/fleet/server/services/preconfiguration.ts:202:40)\n    at createSetupSideEffects (/var/lib/buildkite-agent/builds/kb-n2-4-spot-ec638bb99f0408cd/elastic/kibana-on-merge/kibana/x-pack/plugins/fleet/server/services/setup.ts:105:5)\n    at awaitIfPending (/var/lib/buildkite-agent/builds/kb-n2-4-spot-ec638bb99f0408cd/elastic/kibana-on-merge/kibana/x-pack/plugins/fleet/server/services/setup_utils.ts:30:20)\n    at /var/lib/buildkite-agent/builds/kb-n2-4-spot-ec638bb99f0408cd/elastic/kibana-on-merge/kibana/x-pack/plugins/fleet/server/plugin.ts:415:9"
  },
  "log": {
    "level": "WARN",
    "logger": "plugins.fleet"
  },
  "process": {
    "pid": 2757
  }
}
...
Setup failed: {
  "level": "available",
  "summary": "Fleet setup failed",
  "meta": {
    "error": "[Elastic Cloud agent policy 0001] could not be added. [fleet_server] could not be installed due to error: [PackageNotFoundError: [fleet_server] package not found in registry]"
  }
}

Do you think there is more we can do to avoid this @jen-huang? Is there still an external dependency here? I thought we had the registry running locally now. Can packages still just disappear randomly?

jen-huang commented 2 years ago

@spalger We do have registry running locally but it pulls from docker.elastic.co so that would be the external dependency. We do a custom way of spinning up this docker image as it's not from within functional tests runner: https://github.com/elastic/kibana/blob/95d3090f1ae714ffdf6d239e6fad4aeabe29d924/x-pack/plugins/fleet/server/integration_tests/helpers/docker_registry_helper.ts

@nchaulet Two questions:

Could we use any of the docker services in kbn-test instead of rolling our own? https://github.com/elastic/kibana/tree/main/packages/kbn-test/src/functional_test_runner/lib/docker_servers
If not, could we mimic the waitForLogLine: 'package manifests loaded' configuration in our version in conjunction with or replacing our current timeouts and retries?

cc @kpollich

nchaulet commented 2 years ago

@nchaulet Two questions:

Unfortunately the docker servers is really coupled to the mocha test runner
yes we could mimic the waitForLogLine it will be an easy change and probably more robust

jen-huang commented 2 years ago

Flakiness should be better after #136438.

kibanamachine commented 2 years ago

New failure: CI Build - main

kibanamachine commented 1 year ago

New failure: CI Build - 8.5

kibanamachine commented 1 year ago

New failure: CI Build - 8.6

kibanamachine commented 1 year ago

New failure: CI Build - main

kibanamachine commented 1 year ago

New failure: CI Build - main