Loss doesn't decrease in the multiclass case

[anitaschaefer]

I've adapted the hinge loss from the example to be usable for multiclass data. In essence, I've transformed targets into a one-hot matrix and reshaped preds to (n_samples, n_classes) to match the one-hot matrix:

import numpy as np
import torch
from sklearn.datasets import load_breast_cancer, load_iris, load_digits
from sklearn.model_selection import train_test_split
from torch import Tensor
import torch.nn.functional as F
from lightgbm import LGBMClassifier
from booster_objectives import LightGbmObjective

def hinge_loss(preds: Tensor, targets: Tensor) -> Tensor:
    n_classes = len(np.unique(targets))
    n_samples = targets.size()[0]
    preds = torch.reshape(preds, (n_samples, -1))
    preds = torch.clamp(preds, min=1e-7, max=1.0)
    targets = F.one_hot(targets.long(), n_classes).float()

    targets = 2 * targets - 1
    loss = torch.max(Tensor([0]), 1 - preds * targets) ** 2
    loss = loss.sum()
    return loss

I've used the iris dataset instead and trained the model as usual:

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
custom_objective = LightGbmObjective(loss_function=hinge_loss)
model = LGBMClassifier(objective=custom_objective,
                       n_estimators=10, random_state=42)
model.fit(X_train, y_train, eval_metric="multi_error",
          eval_set=(X_train, y_train), verbose=True)

But the error doesn't decrease during training, instead looking like this: [1] training's multi_error: 1 training's multi_logloss: 34.5388 [2] training's multi_error: 1 training's multi_logloss: 34.5388 [3] training's multi_error: 1 training's multi_logloss: 34.5388 [4] training's multi_error: 1 training's multi_logloss: 34.5388 [5] training's multi_error: 1 training's multi_logloss: 34.5388 [6] training's multi_error: 1 training's multi_logloss: 34.5388 [7] training's multi_error: 1 training's multi_logloss: 34.5388 [8] training's multi_error: 1 training's multi_logloss: 34.5388 [9] training's multi_error: 1 training's multi_logloss: 34.5388 [10] training's multi_error: 1 training's multi_logloss: 34.5388

The binary example works just fine. What may be the cause of this?

[TomerRonen34]

Hi Anita, I belive that multiclass objectives require explicit multiclass code, that I didn't develop. Check out what the LGBM docs say regarding custom losses: https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html#lightgbm.LGBMClassifier It looks like a multiclass LGBM model should already give you 2D preds, without you having to reshape them. If you do add that feature, please let me know, I would be happy to add it to the package 🙂

[Lydia-z]

@anitaschaefer Hi, I am also trying to deal with a problem to implement custom loss function for a multi-class classification problem with LightGBM. Do you have some further solution on this problem?

[anitaschaefer]

@Lydia-z yes, I opened an issue on the lightgbm repository as well, here is the response I got: https://github.com/microsoft/LightGBM/issues/5839#issuecomment-1515403394