ValueError: Classification metrics can't handle a mix of unknown and binary targets

[otoofim]

Hi,

I am using the following callback to get accuracy at the end of each epoch:

('valid_acc', EpochScoring(
'accuracy',
name='valid_acc',
lower_is_better=False,)),

However, at the time of evaluation on the validation set I am getting the error below:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-266-e4ca586adaee> in <module>
      1 get_ipython().run_line_magic('pdb', 'on')
----> 2 gs.fit(X_tra[:100], y_tra[:100])
      3 #%pdb off

c:\users\administrator\pytorch\lib\site-packages\sklearn\model_selection\_search.py in fit(self, X, y, groups, **fit_params)
    720                 return results_container[0]
    721 
--> 722             self._run_search(evaluate_candidates)
    723 
    724         results = results_container[0]

c:\users\administrator\pytorch\lib\site-packages\sklearn\model_selection\_search.py in _run_search(self, evaluate_candidates)
   1189     def _run_search(self, evaluate_candidates):
   1190         """Search all candidates in param_grid"""
-> 1191         evaluate_candidates(ParameterGrid(self.param_grid))
   1192 
   1193 

c:\users\administrator\pytorch\lib\site-packages\sklearn\model_selection\_search.py in evaluate_candidates(candidate_params)
    709                                for parameters, (train, test)
    710                                in product(candidate_params,
--> 711                                           cv.split(X, y, groups)))
    712 
    713                 all_candidate_params.extend(candidate_params)

c:\users\administrator\pytorch\lib\site-packages\sklearn\externals\joblib\parallel.py in __call__(self, iterable)
    915             # remaining jobs.
    916             self._iterating = False
--> 917             if self.dispatch_one_batch(iterator):
    918                 self._iterating = self._original_iterator is not None
    919 

c:\users\administrator\pytorch\lib\site-packages\sklearn\externals\joblib\parallel.py in dispatch_one_batch(self, iterator)
    757                 return False
    758             else:
--> 759                 self._dispatch(tasks)
    760                 return True
    761 

c:\users\administrator\pytorch\lib\site-packages\sklearn\externals\joblib\parallel.py in _dispatch(self, batch)
    714         with self._lock:
    715             job_idx = len(self._jobs)
--> 716             job = self._backend.apply_async(batch, callback=cb)
    717             # A job can complete so quickly than its callback is
    718             # called before we get here, causing self._jobs to

c:\users\administrator\pytorch\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py in apply_async(self, func, callback)
    180     def apply_async(self, func, callback=None):
    181         """Schedule a func to be run"""
--> 182         result = ImmediateResult(func)
    183         if callback:
    184             callback(result)

c:\users\administrator\pytorch\lib\site-packages\sklearn\externals\joblib\_parallel_backends.py in __init__(self, batch)
    547         # Don't delay the application, to avoid keeping the input
    548         # arguments in memory
--> 549         self.results = batch()
    550 
    551     def get(self):

c:\users\administrator\pytorch\lib\site-packages\sklearn\externals\joblib\parallel.py in __call__(self)
    223         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    224             return [func(*args, **kwargs)
--> 225                     for func, args, kwargs in self.items]
    226 
    227     def __len__(self):

c:\users\administrator\pytorch\lib\site-packages\sklearn\externals\joblib\parallel.py in <listcomp>(.0)
    223         with parallel_backend(self._backend, n_jobs=self._n_jobs):
    224             return [func(*args, **kwargs)
--> 225                     for func, args, kwargs in self.items]
    226 
    227     def __len__(self):

c:\users\administrator\pytorch\lib\site-packages\sklearn\model_selection\_validation.py in _fit_and_score(estimator, X, y, scorer, train, test, verbose, parameters, fit_params, return_train_score, return_parameters, return_n_test_samples, return_times, return_estimator, error_score)
    526             estimator.fit(X_train, **fit_params)
    527         else:
--> 528             estimator.fit(X_train, y_train, **fit_params)
    529 
    530     except Exception as e:

c:\users\administrator\pytorch\lib\site-packages\skorch\net.py in fit(self, X, y, **fit_params)
    840             self.initialize()
    841 
--> 842         self.partial_fit(X, y, **fit_params)
    843         return self
    844 

c:\users\administrator\pytorch\lib\site-packages\skorch\net.py in partial_fit(self, X, y, classes, **fit_params)
    799         self.notify('on_train_begin', X=X, y=y)
    800         try:
--> 801             self.fit_loop(X, y, **fit_params)
    802         except KeyboardInterrupt:
    803             pass

c:\users\administrator\pytorch\lib\site-packages\skorch\net.py in fit_loop(self, X, y, epochs, **fit_params)
    754                 self.notify('on_batch_end', X=Xi, y=yi_res, training=False, **step)
    755 
--> 756             self.notify('on_epoch_end', **on_epoch_kwargs)
    757         return self
    758 

c:\users\administrator\pytorch\lib\site-packages\skorch\net.py in notify(self, method_name, **cb_kwargs)
    285         getattr(self, method_name)(self, **cb_kwargs)
    286         for _, cb in self.callbacks_:
--> 287             getattr(cb, method_name)(self, **cb_kwargs)
    288 
    289     # pylint: disable=unused-argument

c:\users\administrator\pytorch\lib\site-packages\skorch\callbacks\scoring.py in on_epoch_end(self, net, dataset_train, dataset_valid, **kwargs)
    415 
    416         with cache_net_infer(net, self.use_caching, y_pred) as cached_net:
--> 417             current_score = self._scoring(cached_net, X_test, y_test)
    418 
    419         self._record_score(net.history, current_score)

c:\users\administrator\pytorch\lib\site-packages\skorch\callbacks\scoring.py in _scoring(self, net, X_test, y_test)
    122             y_test=y_test,
    123             scorer=scorer,
--> 124             is_multimetric=False,
    125         )
    126         return scores

c:\users\administrator\pytorch\lib\site-packages\sklearn\model_selection\_validation.py in _score(estimator, X_test, y_test, scorer, is_multimetric)
    608             score = scorer(estimator, X_test)
    609         else:
--> 610             score = scorer(estimator, X_test, y_test)
    611 
    612         if hasattr(score, 'item'):

c:\users\administrator\pytorch\lib\site-packages\sklearn\metrics\scorer.py in __call__(self, estimator, X, y_true, sample_weight)
     96         else:
     97             return self._sign * self._score_func(y_true, y_pred,
---> 98                                                  **self._kwargs)
     99 
    100 

c:\users\administrator\pytorch\lib\site-packages\sklearn\metrics\classification.py in accuracy_score(y_true, y_pred, normalize, sample_weight)
    174 
    175     # Compute accuracy for each possible representation
--> 176     y_type, y_true, y_pred = _check_targets(y_true, y_pred)
    177     check_consistent_length(y_true, y_pred, sample_weight)
    178     if y_type.startswith('multilabel'):

c:\users\administrator\pytorch\lib\site-packages\sklearn\metrics\classification.py in _check_targets(y_true, y_pred)
     79     if len(y_type) > 1:
     80         raise ValueError("Classification metrics can't handle a mix of {0} "
---> 81                          "and {1} targets".format(type_true, type_pred))
     82 
     83     # We can't have more than one value on y_type => The set is no more needed

ValueError: Classification metrics can't handle a mix of unknown and binary targets

My RNN implementation:

class RNN(nn.Module):

    def __init__(self, input_size, enc_hidden_size, dec_hidden_size, LSTM_hidden, LSTM_layer, output_size, dropout_p):

        super(RNN, self).__init__()

        self.enc_hidden_size = enc_hidden_size

        self.dec_hidden_size = dec_hidden_size

        self.LSTM_hidden_size = LSTM_hidden

        self.enc = nn.Linear(input_size , enc_hidden_size).cuda()

        self.lstm = nn.LSTM(input_size = enc_hidden_size, hidden_size = LSTM_hidden, num_layers = LSTM_layer,
                            dropout = dropout_p, bidirectional = True).cuda()

        self.dec = nn.Linear(LSTM_hidden * 2 , dec_hidden_size).cuda()

        self.output = nn.Linear(dec_hidden_size , output_size).cuda()

        self.dropout = nn.Dropout(p = dropout_p).cuda()

        self.softmax = nn.LogSoftmax(dim=2).cuda()

    def one_step_forward(self, input, hidden):

        encoded = self.enc(input)

        lstm_out, hidden_t1 = self.lstm(encoded.unsqueeze(0), hidden)

        dec = self.dec(lstm_out)

        if self.training:
            dec = self.dropout(dec)

        output = self.output(dec)
        softmax = self.softmax(output)

        return softmax, hidden_t1

    def initHidden(self, batch_size, num_lstm_layers = 2):
        return (torch.zeros(2 * num_lstm_layers, batch_size, self.LSTM_hidden_size).to(device=device), torch.zeros(2 * num_lstm_layers, batch_size, self.LSTM_hidden_size).to(device=device))

My Trainer class:

class Trainer(skorch.NeuralNet):

    def __init__(self, optimizer = torch.optim.Adam,  *args, **kwargs):

        self.optimizer_adam = optimizer
        super().__init__(*args, **kwargs)

    def initialize_optimizer(self, triggered_directly=True):

        kwargs = self._get_params_for('optimizer')
        self.optimizer_adam_ = self.optimizer_adam(self.module_.parameters(), **kwargs)

    def train_step(self, Xi, yi):

        hid = self.module_.initHidden(Xi.size()[0])
        self.module_.zero_grad()
        for i in range(Xi.size()[1]):
            output, hid = self.module_.one_step_forward(Xi[:,i,:].type(torch.FloatTensor).to(device=device), hid)        

        _, predicted = torch.max(output.data, 2)

        correct = (predicted.squeeze(0) == yi.type(torch.LongTensor).to(device=device)).sum()
        accuracy = correct
        loss = self.get_loss(output.squeeze(0), yi.type(torch.LongTensor).to(device = device).squeeze(0), X = Xi, training = True)
        if self.module_.training:
            loss.backward()
            self.optimizer_adam_.step()

        return {'loss':  loss, 'y_pred': output, 'acc': accuracy}

    def infer(self, Xi, yi=None):

        hid = self.module_.initHidden(Xi.size()[0])
        self.module_.eval()
        Xi = skorch.utils.to_tensor(Xi, device = device)
        for i in range(Xi.size()[1]):
            output, hid = self.module_.one_step_forward(Xi[:,i,:].type(torch.FloatTensor).to(device=device), hid)
        self.module_.train()
        return output

    def get_loss(self, y_pred, y_true, **kwargs):

        if len(y_pred.size()) > 2:
            y_pred = y_pred.squeeze(0)
            y_true = y_true.squeeze(0)

        y_true = y_true.squeeze(0)
        y_true = torch.tensor(y_true.long()).to(device=device)

        return super().get_loss(y_pred, y_true, **kwargs)

    def _predict(self, X, most_probable=True):

        # return either predicted word probabilities or the most probable 
        # word using argmax.
        y_probas = []
        for yp in self.forward_iter(X, training=False):
            if most_probable:
                pad = skorch.utils.to_numpy(yp.max(-1)[-1])
            else:
                pad = skorch.utils.to_numpy(yp)
            y_probas.append(pad)
        y_proba = np.concatenate(y_probas, 0)
        return y_proba

    def predict_proba(self, X):
        return self._predict(X, most_probable=False)

    def predict(self, X):
        return self._predict(X, most_probable=True)

    def score(self, X, y):
        y_pred = self.predict(X)

        y_true = y_pred.copy()

        for i, yi in enumerate(y):
            yi = skorch.utils.to_numpy(yi.squeeze())
            y_true[i] = yi
        acc = sklearn.metrics.accuracy_score(y_true.flatten(), y_pred.flatten())
        return acc

Trainer object:

trainer = Trainer(

    criterion = nn.NLLLoss,

    optimizer = torch.optim.Adam,
    optimizer__lr = learning_rate,
    optimizer__weight_decay = weight_decay,

    module = RNN,
    module__input_size = n_features,
    module__enc_hidden_size = enc_hidden_size,
    module__dec_hidden_size = dec_hidden_size,
    module__LSTM_hidden = LSTM_hidden,
    module__LSTM_layer = LSTM_layer,
    module__output_size = output_size,
    module__dropout_p = dropout_p,

    #train_split=None,

    iterator_train__batch_size = BATCH_SIZE,
    iterator_valid__batch_size = BATCH_SIZE,

    max_epochs = epoch,

    callbacks = [skorch.callbacks.ProgressBar(batches_per_epoch='auto'),
                 #epoch_acc,
                 ('valid_acc', EpochScoring(
                'accuracy',
                name='valid_acc',
                lower_is_better=False,)),
                 #skorch.callbacks.EarlyStopping(monitor = 'valid_accuracy'),
                 #skorch.callbacks.EpochScoring(scoring='accuracy', lower_is_better=False),
                 #skorch.callbacks.EpochScoring(scoring='accuracy', on_train=True, lower_is_better=False),
                 #skorch.callbacks.EpochScoring(scoring='balanced_accuracy', lower_is_better=False),
                 #skorch.callbacks.EpochScoring(scoring='f1', lower_is_better=False),
                 #skorch.callbacks.EpochScoring(scoring='f1_micro', lower_is_better=False),
                 #skorch.callbacks.EpochScoring(scoring='f1_macro', lower_is_better=False),
                 #skorch.callbacks.EpochScoring(scoring='precision', lower_is_better=False),
                 #skorch.callbacks.EpochScoring(scoring='recall', lower_is_better=False)
                ],

    device = ('cuda' if use_cuda else 'cpu'),

)

[BenjaminBossan]

Without data, this issue is hard to debug. Searching for the error message (which comes from sklearn), the most likely cause is that either your target or your predictions have wrong values. Make sure that your targets are integers from 0 to n-1 and that your y_proba consists of n classes.

[BenjaminBossan]

@otoofim Any updates on this? If not, I will assume that this issue was caused by the things in my previous post and close it.

[otoofim]

I haven't solved the problem, yet. I am using GridSearchCV with cv = 5. It evaluates the model with no problem on validation set at the end of each epoch and print the results at the end of training but when I use the callback to print the val and training acc during training I am getting the error.

[markfilan]

accuracy_score is a classification metric, you cannot use it for a regression problem. You get the error because these regression model do not produce binary outcomes, but continuous (float) numbers (as all regression model do); so, when scikit-learn attempts to calculate the accuracy by comparing a binary number (true label) with a float (predicted value), it not unexpectedly gives an error. And this cause is clearly hinted at the error message itself:

The sklearn.metrics.accuracy_score(y_true, y_pred) method defines y_pred as:

y_pred : 1d array-like, or label indicator array / sparse matrix. Predicted labels, as returned by a classifier.

Which means y_pred has to be an array of 1's or 0's (predicated labels). They should not be probabilities.