Closed statcom closed 4 years ago
That is definitely a bug. Was able to reproduce it on my end, and it seems like calling score was actually refitting the preprocessing instead of just transforming the data, which I don't think you would want to do in general (for a smaller validation set, this likely results in a bad/incorrect fit). Made a quick change to disable this behaviour by default (I think it got introduced accidentally during some previous changes). Fixed the issue on my toy example.
Fast & sweet! I found it's working now. Thanks!
I am using hyperopt-sklearn to optimize hyperparameters for PCA+SVM. The optimization seemed to run fine with validation error near zero. But when I predicted "validation" and "test" data, the accuracies were about 0.1. I know this is wrong because the validation data was a part of training data. I didn't have this problem without preprocessing. What am I doing wrong?