Proposal:
A transfer learning functionality in NeuralForecast core which handles a transfer learning process automatically to perform transfer learning and/or potential change to cross_validate to cross-validate transfer learning from training on one or multiple datasets while performing cross-validation on one or multiple hold-out datasets.
Another nice-to-have for this would be some way to validate which datasets exactly has been trained on for transparency, such as a property "fitted_datasets" which just contains the unique_ids of the datasets used (this might exist already?).
Use case
Transfer learning has been shown to be quite useful in decreasing forecasting error scores and accuracy gains in other areas now from a research perspective; it is essential to validate the utility of the transfer learning process through cross-validation; this is currently not well supported in the NeuralForecast core methods.
However, there are caveats of which this process does not support cross validation with certain configurations, in the real world, we need to ensure the transfer learning process is worth it through corss validation rather than predicting a single timestep as is done in the tutorial above.
It is impossible to first Fit an instantiated instance of NeuralForecast and use the cross_validation functionality; this is essential if you want to cross-validate the transfer learning utility onto a hold-out dataset, for example. The potential code for this process is below. This will re-train the model, which is not intended (for this transfer learning process) and loses the first fit, essentially just fitting on the hold-out dataset.
master_df = #Dataframe with lots of series
nf = NeuralForecast(models=models, freq='15min', local_scaler_type='robust-iqr')
nf.fit(master_df)
dataset = #a hold out dataset I am using to validate the transfer learning on
cv_df = nf.cross_validation(dataset, use_init_models=False, n_windows=None, val_size=val_set_size, test_size=test_set_size, refit=False)
A potential fix for this issue, of setting all models max steps to 0 was mentioned but this will result in an error down the line because of Pytorch lightning
nf.models[0].max_steps = 0
error:
packages\pytorch_lightning\loops\training_epoch_loop.py", line 442, in _should_check_val_fx is_val_check_batch = (current_iteration + 1) % self.trainer.val_check_batch == 0 ZeroDivisionError: integer division or modulo by zero
If using local_scaler_type, then this process will not work again due to the mismatch in the number of series in the datasets, for example, if the master_df in the code snippet above has 131 series and I want to validate the transfer learning error score on a hold out validation set, the below error will be present.
ValueError: Number of groups in the scaler (131) doesn't match the ones in data (1).
Using scalars, especially with neural networks and high dimensionality series is quite essential.
elephaint
commented
2 months ago
Description
Proposal: A transfer learning functionality in NeuralForecast core which handles a transfer learning process automatically to perform transfer learning and/or potential change to cross_validate to cross-validate transfer learning from training on one or multiple datasets while performing cross-validation on one or multiple hold-out datasets.
Another nice-to-have for this would be some way to validate which datasets exactly has been trained on for transparency, such as a property "fitted_datasets" which just contains the unique_ids of the datasets used (this might exist already?).
Use case
Transfer learning has been shown to be quite useful in decreasing forecasting error scores and accuracy gains in other areas now from a research perspective; it is essential to validate the utility of the transfer learning process through cross-validation; this is currently not well supported in the NeuralForecast core methods.
Transfer learning is quite an involved process to transfer training from some data and validate it on another dataset. A potential process to do this transfer learning process is outlined in a tutorial https://nixtlaverse.nixtla.io/neuralforecast/docs/tutorials/transfer_learning.html
However, there are caveats of which this process does not support cross validation with certain configurations, in the real world, we need to ensure the transfer learning process is worth it through corss validation rather than predicting a single timestep as is done in the tutorial above.
A potential fix for this issue, of setting all models max steps to 0 was mentioned but this will result in an error down the line because of Pytorch lightning
nf.models[0].max_steps = 0error:
packages\pytorch_lightning\loops\training_epoch_loop.py", line 442, in _should_check_val_fx is_val_check_batch = (current_iteration + 1) % self.trainer.val_check_batch == 0 ZeroDivisionError: integer division or modulo by zeroValueError: Number of groups in the scaler (131) doesn't match the ones in data (1).Using scalars, especially with neural networks and high dimensionality series is quite essential.