search

h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
http://h2o.ai
Apache License 2.0
6.87k stars 2k forks source link

Break current tests into categories of purpose #14908

Open exalate-issue-sync[bot] opened 1 year ago

exalate-issue-sync[bot] commented 1 year ago

Currently tests are largely grouped by client and size. Tests should be arranged according to purpose. This will help organize the tests by categories that better indicate how often they should be run and by categories that more closely track what a developer might want to check first. Below are category ideas and the thoughts behind them:

Accuracy Tests -numerical comparisons -see PUBDEV-1913

Negative Tests -are bad values caught, and handled in an expected way?

Performance Tests -metrics of "expected" performance should be defined and measured -are there expected metrics for loaded machines or is this only for unloaded systems?

Degradation Tests -test reaction to slow resources (network and filesystems) -test for memory leaks

Client Tests -does client handle basic REST IO as expected -is Sparkling water in this?

Crash/Hang tests -use dynamic tests to find corner cases

Feature tests -do the features work as expected?

White Box Injection -code based error injection to test finaly{} blocks

We currently repeat larger tests in more than one client. Client tests should be able to be reduced to basic API tests, and remove the computation. The larger computational tests can become part of the Feature Tests (where the client of choice is immaterial).

While all tests are run before releases, many of these tests need only be run if relevant changes are made to their area (as well as periodically to check for unintended changes). As the test suite grows it isn't possible to sustain continued frequent periodic runs of all or even most tests. This break out helps reduce the need to run large grab bags of tests with every change.

Some of these areas may not have significant tests in them currently, so this also provides a roadmap for areas that may need more attention/tests.

exalate-issue-sync[bot] commented 1 year ago

Raymond Peck commented: +1

DinukaH2O commented 1 year ago

JIRA Issue Migration Info

Jira Issue: PUBDEV-1944 Assignee: Former user Reporter: Brandon Hill State: Open Fix Version: N/A Attachments: N/A Development PRs: N/A