search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.47k stars 8.04k forks source link

[Dataset Quality] FTR tests flakiness and atomicity #184145

Open achyutjhunjhunwala opened 1 month ago

achyutjhunjhunwala commented 1 month ago

Our FTR tests were skipped due to Flakiness here

On further analysing, we can see that it was one it block which caused the issue which could be skipped, but currently our it test blocks are not atomic enough. Hence the whole describe block was skipped.

Current View

We have 1 massive describe block which loads common data in the before call. Then every it block as per its need ingest more data without cleaning it up. Due to this they lose atomicity. Meaning

Imagine having 3 it blocks -

it1 it2 it3

Now if someone changes the order -

it1 it3 it2

The tests will start breaking

This is also applicable that if the middle it block is skipped, the following it blocks will also start failing. Due to this, there is a high chance, that the whole describe block would be skipped. This would reduce our coverage and could leave us blind for some time, until we fix it.

Possible Solutions

  1. it blocks should be completely atomic. This means they could be reordered, skipped, without affecting any other it blocks.

  2. If one or more it blocks need to share data, they should be grouped inside a sub describe block with its own before/after blocks, where the cleaning takes place.

  3. Ingesting and cleaning data on each it block could also be time consuming on the CI. Hence to tackle this, a carefully thought through mock data can be prepared which can cater multiple tests. For Example - We can create 3 datasets, with 3 different data and that can cater different it blocks.

  4. Idea must be to make the it blocks as atomic as possible avoiding any data leakage.

  5. Any installed package must be removed in the scope of the it block itself or if multiple it blocks plan to install different package, they can either share a sub describe block where the uninstallation happen.

  6. FTR tests needs to validate the behaviour on UI. This means the tests must focus on UI actions. We have tons of tests where we generate data and load the table and see if what data it has. Then we generate more data and then again refresh the page to see if the Table changed. This is not the Expectations of FTR tests. This is the expectation of API tests that ingestion of Data should change outcome. FTR tests should ideally load a big data once and then handle UI actions mostly.

elasticmachine commented 1 month ago

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)