set up Jenkins jobs for nightlies in non-shared way to track test performance

h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

Apache License 2.0

6.9k stars 2k forks source link

Now that Jeff tweaked Jenkins to measure test performance we need to start tracking the results on a non-shared slave infrastructure. Is there a way to say that certain jobs can't be run with others, or do we need a completely separate cluster?

In any event, this should be done on the nightly builds, and the tests should be run as many times as possible before the next nightly is available, to test speed consistency. This same setup can be used for intermittent detection.

In addition, this should use both the multi-JVM-on-a-box setup as well as Jeff's multi-machine setup (on 10G), so we can also see which code runs significantly slower on multi-machine.

See https://0xdata.atlassian.net/browse/PUBDEV-2178 and https://0xdata.atlassian.net/browse/HEXDEV-474

h2oai / h2o-3

set up Jenkins jobs for nightlies in non-shared way to track test performance #15090