search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.69k stars 8.12k forks source link

Automation Plan: Kibana Visual Testing #33817

Open liza-mae opened 5 years ago

liza-mae commented 5 years ago

This describes the initial test plan and infrastructure work for Kibana visual testing. Evaluating using a commercial tool for visual testing. Also researching cypress.io for visual testing to see if it helps with test flakiness.

We currently have close to 800 functional tests in Kibana repo for selenium and x-pack.

I went through build statistics to see how many runs per month over the last year.

Month Build Runs
Jan 2018 1422
Feb 2018 983
Mar 2018 1517
Apr 2018 1530
May 2018 2408
Jun 2018 2001
Jul 2018 3180
Aug 2018 2999
Sep 2018 2609
Oct 2018 4008
Nov 2018* 1171
Dec 2018 3870
Jan 2019 5576
Feb 2019 6242

I went through existing test count on latest kibana repo master, excluding tests for: intake, plugin, interpreter, graph, home, getting started, context, dev tools, management, logstash pipeline, rollup, watcher, security, API, upgrade, spaces.

Existing test count should be filtered down further for what will be part of a visual test suite and we need to add any missing coverage. We can group different types of visualizations into a dashboard or different canvas elements into one workpad for consolidated checks to reduce number of tests.

Functional Area Existing Tests Visual Tests New Tests Total Eng
*Dashboard 177
*Visualize 223 @liza-mae
*Canvas 3 @liza-mae
*Timelion 7
*Discover 75
*Maps 72 @nreese
Uptime 2 @justinkambic
*Reporting 15
Stack Monitoring 70 @chrisronline
Infrastructure 2 @weltenwort
Logs 1 @weltenwort
*Localization - @liza-mae
Machine Learning -
APM -
Elastic Charts - @markov00
Canvas - @clintandrewhall

(More than one engineer can be assigned to work on a functional area and can be directly or indirectly involved in the test dev) (*feature means it is owned by Kibana team )

Estimating snapshot checks: (No of tests) x (3 browsers available) x (No of runs) x (growth factor) Only Chrome and Firefox currently available in commercial tools, IE to be added later, not sure about Safari.

Test Plan

TBD

liza-mae commented 5 years ago

@spalger would you like to get this PR ready visual testing: https://github.com/elastic/kibana/pull/32953 or would you like us to help?

cc: @wayneseymour @dmlemeshko

liza-mae commented 5 years ago

This should be a separate test suite, need to figure out organization.

liza-mae commented 5 years ago

Development of features for visual testing has been available/open to add tests across each Kibana functional area for a few months now, at that time we had PR checking limited to only visual testing PRs, however now we have decided to enable visual testing against every PR as part of #42989

I have been running stability tests for Percy with our existing tests and so far Canvas and Maps seem pretty stable. There are still issues I am investigating with the other areas, more updates to follow.