Predicting probability for all models

mljar / mljar-supervised

Python package for AutoML on Tabular Data with Feature Engineering, Hyper-Parameters Tuning, Explanations and Automatic Documentation

https://mljar.com

MIT License

3k stars 401 forks source link

Predicting probability for all models #24

Closed agosiewska closed 5 years ago

agosiewska commented 5 years ago

Hello, great AutoML tool! But I have encountered a problem.

When the final model is RF, predict function returns classes (values are equal to 0 or 1). But for other models (CatBoost, Xgboost, LightGBM, NN) , predict function returns probabilities. Is it possible to get probabilities when the best model is Random Forest?

pplonski commented 5 years ago

Hey Alicja! Thanks for reporting. Could you please provide code sample? Might be a bug!

agosiewska commented 5 years ago

Sorry for bothering you, that was my fault. I was testing on a training data, so Random Forest was overfitted and it returned probabilities, but equal 0 and 1. I will close this issue.

pplonski commented 5 years ago

OK :) Have you been able to get a good model?

agosiewska commented 5 years ago

Yes and no :) I was trying to beat h2o AutoML, but results for both frameworks look almost the same.

1h time limit AUC was:
- h2o: 0.7883
- MLJAR: 0.7818
8h time limit:
- h2o: 0.7861
- MLJAR: 0.783

I think maybe it is impossible to achieve more on this dataset. I was training model on FICO data from this XAI competition.

Thank you for developing this AutoML framework, I will definitely be using it in the future!