Quantile Loss Or Pinball loss not properly implemented

[TheMrguiller]

Hi @Jianguo99,

While i was using your amazing tool, i come up with an error in the generation of the loss. It seems that your implementation is not correct. It gives a sum of the losses instead of the expected mean. Please if you have time check it. I am not an expert but comparing to another implementation on the web I observe quite a different result.

class PinballLoss(nn.Module):
    """
    Calculates the quantile loss function.

    Attributes
    ----------
    quantiles : torch.tensor
        The quantiles for which the pinball loss is calculated.
    """
    def __init__(self, quantiles):
        super(PinballLoss, self).__init__()
        self.quantiles = quantiles

    def forward(self, pred, target):
        """
        Computes the loss for the given prediction.

        Parameters
        ----------
        pred : torch.tensor
            Predictions.
        target : torch.tensor
            Target to predict.

        Returns
        -------
        torch.tensor
            The computed pinball loss.
        """
        quantiles = self.quantiles.to(pred.device)
        error = target - pred
        losses = torch.max(quantiles * error, (quantiles - 1) * error)
        loss = torch.mean(losses)
        return loss

For further reference https://discuss.pytorch.org/t/implementing-quantile-loss/131051 and https://github.com/Javicadserres/wind-production-forecast/blob/28310d7dab7b47d7db3d690580505c1a456e471b/src/model/losses.py.

Thank you for your amazing work

[Jianguo99]

Dear TheMrguiller,

Thank you for your question. We have provided an official link for CQR [1], where the QuantileLoss method involves summing first and then averaging.

Please let me know if you have any further questions or if there is anything else I can assist you with.

Best regards, authors

[1] https://github.com/yromano/cqr/blob/master/cqr/torch_models.py

[TheMrguiller]

Hi, Thank you so match for redirecting me to the official implementation. Have tried for an especific case the original implementation and the provided by you and I get different results.

Initial tensor result obtained for the first step of the model:

pred=torch.tensor([[0.4570, 0.3787],
        [0.4851, 0.3821],
        [0.4358, 0.3870],
        [0.4563, 0.3718],
        [0.4736, 0.3784],
        [0.4897, 0.3928],
        [0.4990, 0.3752],
        [0.4666, 0.3433],
        [0.4900, 0.3533],
        [0.4502, 0.4080],
        [0.4727, 0.3967],
        [0.4658, 0.3760],
        [0.4700, 0.3638],
        [0.4827, 0.3865],
        [0.4380, 0.4202],
        [0.4451, 0.3772],
        [0.4573, 0.3914],
        [0.4744, 0.3816],
        [0.4688, 0.4185],
        [0.4729, 0.3750],
        [0.4727, 0.3999],
        [0.4604, 0.4060],
        [0.4907, 0.3933],
        [0.4370, 0.4141],
        [0.4888, 0.3889],
        [0.3979, 0.4214],
        [0.4373, 0.4138],
        [0.4529, 0.4160],
        [0.4592, 0.3845],
        [0.4863, 0.4351],
        [0.4556, 0.3745],
        [0.4521, 0.3821],
        [0.4573, 0.4048],
        [0.4575, 0.3752],
        [0.4338, 0.4163],
        [0.4517, 0.4084],
        [0.4624, 0.3564],
        [0.4490, 0.4028],
        [0.5073, 0.3694],
        [0.4285, 0.3655],
        [0.4434, 0.3970],
        [0.4836, 0.4014],
        [0.4646, 0.3572],
        [0.4834, 0.3560],
        [0.4414, 0.3684],
        [0.4766, 0.3562],
        [0.4954, 0.3828],
        [0.4473, 0.3984],
        [0.4453, 0.4456],
        [0.4941, 0.4094],
        [0.4419, 0.3784],
        [0.4482, 0.3813],
        [0.4573, 0.4001],
        [0.4646, 0.3936],
        [0.4463, 0.3877],
        [0.4397, 0.3884],
        [0.4626, 0.3860],
        [0.4661, 0.3452],
        [0.4319, 0.3662],
        [0.4790, 0.3879],
        [0.4512, 0.3840],
        [0.4685, 0.3828],
        [0.4536, 0.3596],
        [0.4785, 0.3625]])
target=torch.tensor([0.2927, 0.8000, 1.0000, 0.8684, 1.0000, 0.6000, 0.8000, 0.6000, 0.6531,
        0.8000, 0.7273, 0.8000, 0.8000, 0.8000, 0.3636, 0.4000, 0.6000, 0.4000,
        0.8000, 0.8000, 0.4000, 0.6000, 1.0000, 0.0000, 0.4286, 0.6000, 0.8000,
        0.4000, 0.6000, 0.6000, 0.1905, 1.0000, 0.6000, 0.6000, 0.6000, 1.0000,
        0.8000, 0.4000, 0.6000, 0.6000, 0.4000, 0.9630, 0.5758, 0.6000, 0.6000,
        0.1290, 0.4000, 0.2000, 0.6154, 0.6000, 0.8000, 0.6000, 0.8000, 0.0000,
        0.8000, 0.4000, 1.0000, 0.8000, 0.6333, 0.8000, 0.6000, 0.8750, 0.8000,
        0.8169])

Different methods and their obtained results:

alpha = 0.1
quantiles = [alpha / 2, 1 - alpha / 2]
quantile_loss = AllQuantileLoss(torch.tensor(quantiles))
quantile_loss(pred,target.unsqueeze(1))
# Result tensor(0.3075)
regression_loss = QuantileLoss(quantiles=torch.tensor(quantiles))
regression_loss(pred,target.unsqueeze(1))
# Result tensor(9.8214)

I hope this gives better context to the problem.

[Jianguo99]

Dear TheMrguiller,

I have reviewed the codes of quantile loss in TorchCP and CQR. There appear to be some issues with the "AllQuantileLoss" implementation. The shape of the "errors" tensor in "AllQuantileLoss" is 64*64 instead of the expected 64*1. This discrepancy is likely the main cause of the different numerical results between the two implementations.

I look forward to your response on this matter.

Best regards, Authors

[TheMrguiller]

Sorry, I have check what i send you at it seems my error. The "error" tensor is shaped 64x64 due to my error, the original shape is 64. The code should be:

quantile_loss = AllQuantileLoss(torch.tensor(quantiles))
quantile_loss(pred,target) 

Even so there seems to be a missmatch between the loss errors. Both return the same values as before. And if you look at the range of my values,0-1, it makes sense to be in the number of 0.3. Maybe i am looking wrongly. I also obtain another quantile regression formula, in this case for each quantile:

def quantile_loss_function(preds, target, quantile):
    assert 0 < quantile < 1, "Quantile should be in (0, 1) range"
    errors = target - preds
    loss = torch.max((quantile - 1) * errors, quantile * errors)
    return torch.abs(loss).mean()

first_quantile=quantile_loss_function(pred[:,0],target,torch.tensor(quantiles[0]))
second_quantile=quantile_loss_function(pred[:,1],target,torch.tensor(quantiles[1]))
overall_loss=first_quantile+second_quantile
print(overall_loss)

This one gives nearly the same value as AllQuantileLoss.

[Jianguo99]

Dear TheMrguiller,

Thank you for your reminder. The original "QuantileLoss" ensurly has some miswriting. I have updated it in the branch "development" and will issue a new version of TorchCP in PyPI. You can use the new "Quantileloss" :

regression_loss = QuantileLoss(quantiles=torch.tensor(quantiles))
regression_loss(pred,target)

Best regards, Authors