Open gaoexingyun opened 4 years ago
lucasmejiall
commented
4 years ago I had the same problem. I believe it's this line causing the problem in estimator.py line 333:
if continuous_loss_fn: cv_pred_pool = np.append(cv_pred_pool, learner.predict_proba(XEXval))
It should be:
if continuous_loss_fn: cv_pred_pool = np.append(cv_pred_pool, learner.predict_proba(XEXval)[:, 1])
jhmenke
commented
4 years ago I had the same problem. I believe it's this line causing the problem in estimator.py line 333:
if continuous_loss_fn: cv_pred_pool = np.append(cv_pred_pool, learner.predict_proba(XEXval))It should be:
if continuous_loss_fn: cv_pred_pool = np.append(cv_pred_pool, learner.predict_proba(XEXval)[:, 1])
What is the purpose of this? You would only select a single class probability per sample from predict_proba
linehammer
commented
3 years ago Sounds like the shapes of your labels and predictions are not in alignment. I faced a similar problem while fitting a regression model . The problem in my case was, Number of rows in X was not equal to number of rows in y. You likely get problems because you remove rows containing nulls in X_train and y_train independent of each other. y_train probably has few, or no nulls and X_train probably has some. So when you remove a row in X_train and the same row is not removed in y_train it will cause your data to be unsynced and have different lenghts. Instead you should remove nulls before you separate X and y.
In most case, x as your feature parameter and y as your predictor. But your feature parameter should not be 1D. So check the shape of x and if it is 1D, then convert it from 1D to 2D.
x.reshape(-1,1)
WandrilleD
commented
2 weeks ago 2024 and this issue is still present. I think the solution has been rightfully pointer by @jhmenke above.
I write here a "minimalist" demonstration of the problem, as well as a little fix for people encountering this issue in their code:
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score
from hpsklearn import HyperoptEstimator,svc
X,y = load_breast_cancer(return_X_y=True)
def roc_auc_loss(y_target, y_prediction ):
print( y_target.shape , y_prediction.shape ) ## looking up shapes
print( y_prediction[:10] ) ## looking up what's in the predictions
p0 = y_prediction[::2] ## half of the elements
p1 = y_prediction[1::2] ## other half of the elements
print( (p0+p1)[:10] ) ## what do they sum to?
return -roc_auc_score(y_target, y_prediction)
estim = HyperoptEstimator(classifier=svc("mySVC",probability=True),
loss_fn = roc_auc_loss,
continuous_loss_fn = True,
trial_timeout=10)
estim.fit(X, y)which yields:
(114,)
(228,)
[0.41895505 0.58104495 0.22532898 0.77467102 0.07204063 0.92795937
0.03291461 0.96708539 0.0034852 0.9965148 ]
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
0%| | 0/1 [00:00<?, ?trial/s, best loss=?]
job exception: Found input variables with inconsistent numbers of samples: [114, 228]followed by the error traceback.
As we can see the error stems from the fact the predict_proba() returns an array with 1 column per category, which is flattened and then passed to the scoring function.
We confirm that by checking that we get 1.0 when summing together each odd and even indexed elements.
Here is my quick fix if you don't want to wait for the library to be updated:
def roc_auc_loss_fixed(y_target, y_prediction ):
p1 = y_prediction[1::2] ## half of the elements corresponding to proba of being category 1
return -roc_auc_score(y_target, p1)
estim = HyperoptEstimator(classifier=svc("mySVC",probability=True),
loss_fn = roc_auc_loss_fixed,
continuous_loss_fn = True,
trial_timeout=10)
estim.fit(X, y)
When I use HyperoptEstimator to train the model, I want to calculate logloss, so I manually set several classifiers to support predict_proba, but it will report errorError: Found input variables with inconsistent numbers of samples: [1818794, 79078]. What is the reason?
Following is my code：
from sklearn.model_selection import cross_val_score from sklearn.model_selection import StratifiedKFold from sklearn.metrics import log_loss from hpsklearn import sklearn_RandomForestClassifier,extra_trees,random_forest,knn,svc from hyperopt import hp from random import choice def algorithms(name): classifiers = [
kf = StratifiedKFold(n_splits=5,shuffle=True)
use sigmoid kernal to train model
Hyper_logloss = [] for train_index, test_index in kf.split(X,y): X_train, X_test = X.iloc[train_index], X.iloc[test_index] y_train, y_test = y.iloc[train_index], y.iloc[test_index] estim = HyperoptEstimator(classifier=algorithms("clf"),algo=tpe.suggest,loss_fn=log_loss,continuous_loss_fn=True,trial_timeout=3600) estim.fit(X_train, y_train)
ValueError Traceback (most recent call last)