Failing test: X-Pack Endpoint Functional Tests.x-pack/test/security_solution_endpoint/apps/endpoint - endpoint "before all" hook in "endpoint"

[kibanamachine]

A test failed on a tracked branch

Error: expected 200 "OK", got 500 "Internal Server Error"
    at Test._assertStatus (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:268:12)
    at Test._assertFunction (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:283:11)
    at Test.assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:173:18)
    at assert (/dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:131:12)
    at /dev/shm/workspace/kibana/node_modules/supertest/lib/test.js:128:5
    at Test.Request.callback (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/index.js:718:3)
    at parser (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/index.js:906:18)
    at IncomingMessage.res.on (/dev/shm/workspace/kibana/node_modules/superagent/lib/node/parsers/json.js:19:7)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

First failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-test-triage (failed-test)

[kibanamachine]

New failure: Jenkins Build

[elasticmachine]

Pinging @elastic/endpoint-management (Team:Endpoint Management)

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[jen-huang]

The two failures were due to registry connection issues. #72621 should fix that.

[kibanamachine]

New failure: CI Build - 8.0

[kibanamachine]

New failure: CI Build - 8.0

[spalger]

Skipped as this is breaking with the new 8.0 branch:

master/9.0: https://github.com/elastic/kibana/commit/83288af8bd200e607b31d7c50253786a9d1a9382 8.0: https://github.com/elastic/kibana/commit/94ac24692f1996934b76c9378f0c935d05566e24

[jen-huang]

@kevinlog I suspect that this is due to no compatible version of Endpoint package being found once Kibana version is 8.0 (due to conditions.kibana.version field on the package manifest). Any plans to publish a new package soon?

[pzl]

aha, yeah fair point. The current package is restricted with: ^7.16.0 Which is 7.16 and beyond, until 8.0.

We have been doing this for quite some time, (maybe 7.13?). I'm surprised this hasn't come up before on master.

I guess we can cut another package that will begin supporting the 8.0 series.. ?

[jen-huang]

@pzl Yeah, it didn't occur previously on master because of a special condition in Fleet: https://github.com/elastic/kibana/blob/4efabacd4ef59128582089bff1f594658dbf40ed/x-pack/plugins/fleet/server/services/epm/registry/index.ts#L146-L149

I'll assign this to you :)

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kevinlog]

I'm seeing the following logs from the failed test runs that makes it look like Kibana isn't reaching the package registry:

proc [kibana] [2023-02-08T17:01:52.449+00:00][ERROR][plugins.fleet] Failed to fetch latest version of endpoint from registry: Error connecting to package registry: request to http://localhost:6104/search?package=endpoint&prerelease=false&kibana.version=8.7.0 failed, reason: connect ECONNREFUSED 127.0.0.1:6104

...

proc [kibana] [2023-02-08T17:01:56.345+00:00][ERROR][plugins.fleet] Failed to fetch latest version of synthetics from registry: Error connecting to package registry: request to http://localhost:6104/search?package=synthetics&prerelease=true&kibana.version=8.7.0 failed, reason: connect ECONNREFUSED 127.0.0.1:6104

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[jbudz]

/skip

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[jbudz]

Skipped

main: 4c5da503757c35eed6d3ad30379bc2588cece87f

[kevinlog]

Seeing the following in the logs since yesterday:

│ info [o.e.b.BootstrapChecks] [ftr] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Is it related to the failures reported by the tests?

└- ✖ fail: endpoint "before all" hook in "endpoint"
--
  | │      Error: validation_exception
  | │   Caused by:
  | │       search_phase_execution_exception: all shards failed
  | │   Root causes:
  | │       validation_exception: Validation Failed: 1: Failed to test query, received status: SERVICE_UNAVAILABLE;

@joeypoon @paul-tavares - any ideas here?

[kibanamachine]

New failure: CI Build - main

[paul-tavares]

This seems like to me that ES just completely failed. I don't think the tests would have caused that, but I do wonder if the issues with Docker (just saw email) would have caused this problem since we use a docker container to run the Package Registry. Maybe.

PR to unskip and run flaky tests on it here: https://github.com/elastic/kibana/pull/152039

[kibanamachine]

New failure: CI Build - 8.7

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[jbudz]

Skipped

main: 05f556b162f19ccebecb8a2f95c10a3255cfee6c

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - main

[mistic]

Skipped.

main: bb824d3

[kibanamachine]

New failure: CI Build - main

[kibanamachine]

New failure: CI Build - 8.11

[kevinlog]

latest error:

└-: endpoint
--
  | └-> "before all" hook: beforeTestSuite.trigger in "endpoint"
  | └-> "before all" hook in "endpoint"
  | │ERROR Error: resource_not_found_exception
  | │       Root causes:
  | │           resource_not_found_exception: Transform with id [endpoint.metadata_united-default-8.11.0] could not be found
  | │          at updatePackages (setup_fleet_for_endpoint.ts:166:15)
  | │          at processTicksAndRejections (node:internal/process/task_queues:95:5)
  | │          at retryOnError (utils.ts:81:22)
  | │          at Context.<anonymous> (index.ts:35:7)
  | │          at Object.apply (wrap_function.js:73:16)
  | │ERROR retryOnError(): non-retryable error encountered