search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.67k stars 8.23k forks source link

[Security Solution][Serverless] Automation readiness for GA #189468

Open MadameSheema opened 3 months ago

MadameSheema commented 3 months ago

Summary

Before going GA, we need to make sure that all our core functionalities are properly tested and covered with automation, since we are not going to have the same amount of time to do manual testing before releasing to production.

To do so we need to:

  1. Identify the core functionalities of each one of the area teams.
  2. Make sure that those functionalities are covered by automation (Cypress or API tests)
  3. If the functionality is covered by automation:
    • Make sure that the test is not skipped
    • Make sure the test is executed on the periodic pipeline
    • If the test is a candidate for the Kibana QA quality gate add it
  4. If the functionality is not covered by automation:
    • Add automation to it
    • Make sure the test is executed on the periodic pipeline
    • If the test is a candidate for the Kibana QA quality gate add it
elasticmachine commented 3 months ago

Pinging @elastic/security-solution (Team: SecuritySolution)

mchopda commented 3 months ago

Thanks Glo for creating this issue and this to track any missing automation.

michaelolo24 commented 1 month ago

Just an update that we're in the process of un-skipping tests within the security threat hunting cypress code base to restore existing coverage, and then adding additional coverage for any pending/uncovered user journeys. Will update here when the work is completed

banderror commented 1 month ago

We have done certain work to prepare for Serverless GA in terms of test automation (details in https://github.com/elastic/security-team/issues/10609), and all our pipelines have been green. For the Rule Management team this ticket has been addressed. cc @mchopda

jaredburgettelastic commented 1 month ago

EA has one ticket to unskip some asset criticality tests in Serverless here, although the associated functionality has been manually verified.

michaelolo24 commented 1 month ago

Threat hunting has two open PRs for unskipping the majority of our existing integration tests here to get to baseline readiness. They should be merged this week. None of the changes required changes to any security functionality, but rather updates to the tests to account for changes that were made in other packages.

  1. https://github.com/elastic/kibana/pull/195721
  2. https://github.com/elastic/kibana/pull/195662
yctercero commented 1 month ago

Detection Engine:

Identify the core functionalities of each one of the area teams.

✅

Make sure that those functionalities are covered by automation (Cypress or API tests)

✅

If the functionality is covered by automation: Make sure that the test is not skipped

This is forever ongoing as new flakes pop up. We try to prioritize unskipping critical tests. Some have unresolved flakiness and we continue to try to debug. The values in this spreadsheet are probably about 2 weeks old, but give an overview of our tests and where they stand.

Make sure the test is executed on the periodic pipeline

We're pretty good with our FTRs. There are a number of Cypress tests that are skipped due to flakiness.

If the test is a candidate for the Kibana QA quality gate add it

✅

If the functionality is not covered by automation: Add automation to it

Currently working to add FTRs for serverless prebuilt roles.

cc @mchopda

mchopda commented 3 weeks ago

Thanks all. @MadameSheema Could you please comment on the completeness of the issue?

MadameSheema commented 2 weeks ago

Hello @mchopda,

All the teams are putting in great effort to ensure their automation is ready for the serverless GA, and so far, I feel comfortable with our progress.

However, we should consider that this needs to be an ongoing effort. It’s important to ensure that with every new functionality we develop, we have coverage on MKI. Additionally, we need to monitor new skipped tests to ensure coverage is not impacted.

MadameSheema commented 1 week ago

NOTE: This applies only to teams that have FTR (API) tests with a direct dependency on Elasticsearch.

Another action we would like to take before going GA is to add tests to the Elasticsearch Serverless Verify and Promote Pipeline.

This will help prevent Elasticsearch from releasing a version that could break our application.

cc @banderror @yctercero @michaelolo24 @jaredburgettelastic @peluja1012 @dasansol92

MadameSheema commented 1 week ago

Ignore all of the above links, were introduced by a mistake.