Tests using Auto(*) currently always pass as errors are not raised.

alteryx / evalml

EvalML is an AutoML library written in python.

https://evalml.alteryx.com

BSD 3-Clause "New" or "Revised" License

760 stars 86 forks source link

Tests using Auto(*) currently always pass as errors are not raised. #37

Closed jeremyliweishih closed 4 years ago

jeremyliweishih commented 5 years ago

Right now there is no way of telling if the Auto(*) tests actually fail as we're not raising errors when training the pipelines. To solve this we can check for NaN scores or raise errors.

kmax12 commented 5 years ago

the raise_errors parameter may be helpful to set to true during testing

https://evalml.featurelabs.com/en/latest/generated/methods/evalml.AutoClassifier.fit.html#evalml.AutoClassifier.fit

angela97lin commented 4 years ago

Closing for now, addressed by #254 which turned on the raise_errors flag for all current tests... but it could be useful to discuss how to make this process better (set env var so raise_errors is true while testing)?