[Meta] Update serverless test to not run with operator privileges

pheyos commented 2 weeks ago

Overview

The serverless operator user should not be used for functional / e2e testing. There are some reasons for it:

The operator user has more privileges than a regular serverless user would have. So tests running as operator are not exactly covering what a real user would experience.
The operator user is treated differently by some parts of Elasticsearch, e.g. the REST parameter validation. So a test might pass due to these elevated privileges where it should actually fail.

Our UI and API integration tests should be close to the actions a real user would perform in order to have best possible test coverage. This includes user privileges.

Having said all that, the operator user is still required in some places like test setup or teardown. But it should not be used for login nor for the API under test.

Details and examples of how to do that are in the serverless test readme: https://github.com/elastic/kibana/blob/main/x-pack/test_serverless/README.md#roles-based-testing

cc as discussed @cachedout @sphilipse @MindyRS

High level areas to update

Target date is end of June 2024

Tagging the teams to drive it - not necessarily to implement it.

Group 1: `x-pack/test_serverless/api_integration/`

[ ] test_suites/common
- @elastic/appex-qa
- tracking issue: https://github.com/elastic/kibana/issues/180834
[ ] test_suites/observability
- @paulb-elastic
[ ] test_suites/search
- @elastic/enterprise-search-frontend
- tracking issue: https://github.com/elastic/enterprise-search-team/issues/7544
[ ] test_suites/security
- @elastic/security-engineering-productivity

Group 2: `x-pack/test_serverless/functional/`

[ ] test_suites/common
- @elastic/appex-qa
- initial PR: https://github.com/elastic/kibana/pull/176221
[ ] test_suites/observability
- @paulb-elastic
[ ] test_suites/search
- @elastic/enterprise-search-frontend
- tracking issue: https://github.com/elastic/enterprise-search-team/issues/7544
[ ] test_suites/security
- @elastic/security-engineering-productivity

Group 3: `x-pack/test/security_solution*`

[ ] test/security_solution_api_integration (serverless tests)
- @elastic/security-engineering-productivity
[ ] test/security_solution_cypress (serverless tests)
- @elastic/security-engineering-productivity
[ ] test/security_solution_endpoint (serverless tests)
- @elastic/security-engineering-productivity
[ ] test/security_solution_endpoint_api_int (serverless tests)
- @elastic/security-engineering-productivity
[ ] test/security_solution_cypress (serverless tests)
- @elastic/security-engineering-productivity

elasticmachine commented 2 weeks ago

Pinging @elastic/appex-qa (Team:QA)

sphilipse commented 2 weeks ago

Added tracking issue for Search and updated tagged team

paulb-elastic commented 1 week ago

https://github.com/elastic/kibana/issues/184033 created to address the Observability requirements and updated the contact to me (we don't have a corresponding GitHub team, but a project team for this)

MadameSheema commented 1 week ago

In Security we have started implementing the change in our Cypress tests: https://github.com/elastic/kibana/pull/183608 and then we'll move forward with the API ones.

elastic / kibana