Neuralforecast models don't have populated train_trajectories field in v1.6.0

amlimaye commented 1 year ago

What happened + What you expected to happen

I use the model.train_trajectories attribute in order to look at the training loss history of a neuralforecast model. Upon upgrade to neuralforecast 1.6.0, it appears that this attribute is an empty list, even after taking training steps.

Versions / Dependencies

This is a regression from neuralforecast 1.5.0 behavior that seems to have been introduced in 1.6.0

Reproduction script

from neuralforecast.models import NHITS
from neuralforecast.utils import AirPassengersDF

# Split data and declare panel dataset
Y_df = AirPassengersDF
Y_train_df = Y_df[Y_df.ds<='1959-12-31'] # 132 train
Y_test_df = Y_df[Y_df.ds>'1959-12-31'] # 12 test

# Fit and predict with NBEATS and NHITS models
horizon = len(Y_test_df)
models = [NHITS(input_size=2 * horizon, h=horizon, max_steps=50)]
nf = NeuralForecast(models=models, freq='M')
nf.fit(df=Y_train_df)

print(nf.models[0].train_trajectories)

When run with neuralforecast 1.5.0, I see the following output:

Epoch 49: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 24.66it/s, v_num=0, train_loss_step=15.90, train_loss_epoch=15.90]
[(0, 47.025596618652344), (1, 82.90968322753906), (2, 52.398529052734375), (3, 47.62156677246094), (4, 42.74787521362305), (5, 39.8175163269043), (6, 41.74231719970703), (7, 40.752113342285156), (8, 36.911922454833984), (9, 36.49498748779297), (10, 33.973854064941406), (11, 32.00471115112305), (12, 29.488445281982422), (13, 29.370990753173828), (14, 30.641225814819336), (15, 28.497554779052734), (16, 25.36324119567871), (17, 24.675334930419922), (18, 24.729576110839844), (19, 23.2929744720459), (20, 23.000173568725586), (21, 23.211406707763672), (22, 21.795318603515625), (23, 22.2393798828125), (24, 21.003341674804688), (25, 21.002304077148438), (26, 21.38489532470703), (27, 20.440746307373047), (28, 20.385372161865234), (29, 20.317750930786133), (30, 19.628896713256836), (31, 19.111839294433594), (32, 18.837032318115234), (33, 18.77350616455078), (34, 18.57549476623535), (35, 18.199424743652344), (36, 18.268062591552734), (37, 18.0363826751709), (38, 17.5035457611084), (39, 17.451993942260742), (40, 17.490385055541992), (41, 17.647701263427734), (42, 17.30829429626465), (43, 16.701465606689453), (44, 16.762176513671875), (45, 16.510028839111328), (46, 16.256656646728516), (47, 16.200454711914062), (48, 16.48953628540039), (49, 15.918562889099121)]

When run with neuralforecast 1.6.0, I see the following output:

Epoch 49: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 21.79it/s, v_num=1, train_loss_step=15.90, train_loss_epoch=15.90]
[]

Issue Severity

Low: It annoys or frustrates me.

amlimaye commented 1 year ago

Happy to take a shot at fixing this, although I haven't been able to pinpoint which change included in 1.6.0 is the source of this regression.

cchallu commented 1 year ago

Hi @amlimaye. We changed the trained models to the attribute models_fitted, not models. But we will roll-back this change due to the generalized confusion tomorrow. It will be again in models in tomorrow's release.

Nixtla / neuralforecast