[Meta] Update serverless test to not run with operator privileges

pheyos commented 6 months ago

Overview

The serverless operator user should not be used for functional / e2e testing. There are some reasons for it:

The operator user has more privileges than a regular serverless user would have. So tests running as operator are not exactly covering what a real user would experience.
The operator user is treated differently by some parts of Elasticsearch, e.g. the REST parameter validation. So a test might pass due to these elevated privileges where it should actually fail.

Our UI and API integration tests should be close to the actions a real user would perform in order to have best possible test coverage. This includes user privileges.

Having said all that, the operator user is still required in some places like test setup or teardown. But it should not be used for login nor for the API under test.

Details and examples of how to do that are in the serverless test readme: https://github.com/elastic/kibana/blob/main/x-pack/test_serverless/README.md#roles-based-testing

cc as discussed @cachedout @sphilipse @MindyRS

High level areas to update

Target date is end of June 2024

Tagging the teams to drive it - not necessarily to implement it.

Group 1: `x-pack/test_serverless/api_integration/`

[x] test_suites/common
- @elastic/appex-qa
- tracking issue: https://github.com/elastic/kibana/issues/180834
[x] test_suites/observability
- @paulb-elastic
- https://github.com/elastic/kibana/pull/184654
[x] test_suites/search
- @elastic/enterprise-search-frontend
- tracking issue: https://github.com/elastic/enterprise-search-team/issues/7544
[ ] test_suites/security
- @elastic/security-engineering-productivity

Group 2: `x-pack/test_serverless/functional/`

[x] test_suites/common
- @elastic/appex-qa
- initial PR: https://github.com/elastic/kibana/pull/176221
[x] test_suites/observability
- @paulb-elastic
- https://github.com/elastic/kibana/pull/184823
[x] test_suites/search
- @elastic/enterprise-search-frontend
- tracking issue: https://github.com/elastic/enterprise-search-team/issues/7544
[ ] test_suites/security
- @elastic/security-engineering-productivity

Group 3: `x-pack/test/security_solution*`

[ ] test/security_solution_api_integration (serverless tests)
- @elastic/security-engineering-productivity
- https://github.com/elastic/kibana/pull/185870
[x] test/security_solution_cypress (serverless tests)
- @elastic/security-engineering-productivity
- https://github.com/elastic/kibana/pull/183608
[ ] test/security_solution_endpoint (serverless tests)
- @elastic/security-engineering-productivity
[ ] test/security_solution_endpoint_api_int (serverless tests)
- @elastic/security-engineering-productivity
[ ] test/security_solution_cypress (serverless tests)
- @elastic/security-engineering-productivity

elasticmachine commented 6 months ago

Pinging @elastic/appex-qa (Team:QA)

sphilipse commented 6 months ago

Added tracking issue for Search and updated tagged team

paulb-elastic commented 6 months ago

https://github.com/elastic/kibana/issues/184033 created to address the Observability requirements and updated the contact to me (we don't have a corresponding GitHub team, but a project team for this)

MadameSheema commented 6 months ago

In Security we have started implementing the change in our Cypress tests: https://github.com/elastic/kibana/pull/183608 and then we'll move forward with the API ones.

sophiec20 commented 5 months ago

The end-june date is approaching .. updates please @pheyos @cachedout @sphilipse @MindyRS

cachedout commented 5 months ago

Thanks for the bump, @sophiec20 . On the Observability side, this was turned over to @paulb-elastic who tells me that the work has been completed.

paulb-elastic commented 5 months ago

Correct, the Observability items have been completed via the various PRs linked above

MindyRS commented 5 months ago

@sophiec20 Teams on our side have tickets assigned. Some are complete, some in flight, some in backlogs. Everyone is aware of the deadline and we'll have it complete by the end of the month. https://github.com/elastic/kibana/issues/184135

sphilipse commented 5 months ago

Thanks for the reminder @sophiec20, we're looking to get this done before end of June. There's not a lot of work on our end, to be clear.

pheyos commented 5 months ago

Almost done on the Platform side. One last API integration test suite left that is work in progress.

sphilipse commented 5 months ago

Search is done, thanks to @saarikabhasi's work :)

elastic / kibana