Validation and Training Loss

[hugogobato]

What happened + What you expected to happen

I noticed that the training and validation loss shown are based on the y's and y_hat's before the inverse normalization, which makes it difficult to compare models with different types of scaler used by TemporalNorm and to properly plot the training and validation loss per epoch. Thus, I would like to recommend to first perform the inverse normalization and then estimate the training and validation loss.

Versions / Dependencies

neuralforecast-1.6.3-py3-none-any.whl

Reproduction script

For example the code below and for scaler_type=, you can try 'standard', 'minmax', and 'robust' model = NBEATSx(h=12, input_size=24,

loss=MQLoss(level=[80, 90]),

            loss=DistributionLoss(distribution='Normal', level=[80, 90]),
            scaler_type='robust',
            dropout_prob_theta=0.5,
            stat_exog_list=['airline1'],
            futr_exog_list=['trend'],
            max_steps=200,
            val_check_steps=10,
            early_stop_patience_steps=2)

nf = NeuralForecast( models=[model], freq='M' ) nf.fit(df=Y_train_df, static_df=AirPassengersStatic, val_size=12) Y_hat_df = nf.predict(futr_df=Y_test_df)

Issue Severity

High: It blocks me from completing my task.

[cchallu]

Hi @hugogobato. This issue was fixed in #720 in August 15 for point losses. Are you using the latest version? The training loss is not scaled back, but you should still not use it directly for evaluating/selecting models.

For Distribution losses the validation loss is scaled back:

[hugogobato]

@cchallu, thank you very much for the response and information. I was asking for the training loss with the scaled back data to create plots of the training and validation loss per epoch. By having the real data instead of the scaled data could facilitate the interpretability for non-expert stakeholders (for example in my case showing the training and validation loss plots of a model for forecasting stock realized volatility, which can only have positive yet quite small values (x>0), to investors). Additionally, I am working on a loss function specific for forecasting stock realized volatility (which I plan use it to contribute to your awesome project by creating a pull request) that only accepts positive values (x>0), and I am struggling with testing it since it does not work depending on the scaler type that I choose.

[Alisia0182]

Hi @hugogobato, I was wondering if you have figured out how to create plots of the training and validation loss per epoch with the scaled back data. I want to do the same thing but I found it's not supported in the current implementation.

[LZBNCC]

Hi @hugogobato, I was wondering if you have figured out how to create plots of the training and validation loss per epoch with the scaled back data. I want to do the same thing but I found it's not supported in the current implementation.

hello， I was wondering if you have finished the task that create plots of the training and validation loss per epoch with the scaled back data，if have, could you please tell me how to do it?

[evandrocardozo]

@hugogobato i'm also interested in tracking train/validation losses by epoch, but haven't found any support on nixtla's documentation in how to perform this plot. If you could provide your approach, it would be highly appreciated. Thanks!

[jmoralez]

The train loss isn't scaled back, the valid loss is and you can find both either in the model attributes or in the logged files. Here's an example:

from neuralforecast import NeuralForecast
from neuralforecast.models import NBEATS
from neuralforecast.utils import generate_series

series = generate_series(2, min_length=50, max_length=50)
h = 10
nf = NeuralForecast(
    models=[
        NBEATS(
            input_size=h,
            h=h,
            max_steps=20,
            scaler_type='standard',
            enable_progress_bar=False,
            val_check_steps=1,
        )
    ],
    freq='D',
)
nf.fit(series, val_size=h)
train_losses = nf.models[0].train_trajectories
valid_losses = nf.models[0].valid_trajectories

This also produces a directory under lightning_logs with a version (the output can be customized by providing a logger instead). So you can run tensorboard --logdir lightning_logs and visualize those metrics in tensorboard as well.

[evandrocardozo]

@jmoralez Appreciate your feedback! I think that solves what I'm trying to do. Any progress on the scaling issue for the training loss?

Also, is it possible in Nixtla to use the predict method to extend the forecasting over the whole test set length? e.g. train a NeuralForecast model to predict, let`s say, h=7 days ahead, then predict on a test set using futr predictors on a total horizon of 30 days (Kinda like a rolling window method).

Thanks!

[jmoralez]

I think @cchallu was against rescaling the training loss.

For the other thing, can you open a separate issue?