search

adriangb / scikeras

Scikit-Learn API wrapper for Keras.
https://www.adriangb.com/scikeras/
MIT License
239 stars 47 forks source link

Different behavior if param is passed to constructor or in fit() or in set_params() #243

Closed 8549 closed 3 years ago

8549 commented 3 years ago

I noticed that SciKeras' KerasClassifier behaves differently when the parameter class_weight is passed in different ways: if the param is passed in the constructor the model accuracy is way higher than if passing the param in the fit() method. When passing it using set_params() (after instantiating and building the model and before fitting it) the results are similar to when passing in the constructor.

The param is routed (passed as fit__class_weight) when passing in the constructor and via set_params().

You can find a MWE in this Colab notebook.

Is this the intended behavior or is it a bug? I didn't find any note about this in the documentation. Thanks in advance

adriangb commented 3 years ago

Hi @8549, thank you for the report!

This is definitely a bug in SciKeras. The issue boiled down to the default value of class_weight (None) overriding the routed parameter fit__class_weight in the constructor (model2 in your notebook). Because scoring is being done on imbalanced classes, not having a class weighting actually improves the accuracy of the model.

I opened #244 to rectify this, and here is a copy of your notebook showing the change: https://colab.research.google.com/drive/1BpULRmi1N5W5rquliaU8JVd4c4COCG37?usp=sharing

I also used random_state=0 to get more reproducible results (all of the scores are the same).

8549 commented 3 years ago

Thank you so much! Will this fix be included in the next 0.4.1 version? So that i know when to safely update and fix my code.

adriangb commented 3 years ago

yes, in fact it's already released: https://github.com/adriangb/scikeras/releases/tag/v0.4.1

Please let me know if this did not fix it for you!

Also, SciKeras supports class_weight="balanced" so that you don't need to calculate it yourself.