[BUG] RegressionEnsembleModel predict error

[GeorgeXiaojie]

Describe the bug A clear and concise description of what the bug is.

scene1：as below, RegressionEnsembleModel train and predict works

train:

RegressionEnsembleModel(
    forecasting_models = [
        LightGBMModel(
            lags=args.input_length,
            lags_future_covariates=list(range(args.forecast_horizon)),
            # add_encoders={'datetime_attribute': {'future': ['hour', 'minute']}},
            output_chunk_length=args.forecast_horizon,
            random_state=2022,
            force_col_wise=True,
            device='gpu' if args.device != 'cpu' else args.device,
            verbose=1
        ),
        XGBModel(
            lags=args.input_length,
            lags_future_covariates=list(range(args.forecast_horizon)),
            # add_encoders={'datetime_attribute': {'future': ['hour', 'minute']}},
            output_chunk_length=args.forecast_horizon,
            random_state=2022
        )],
    regression_train_n_points =args.forecast_horizon
)

model.fit(
    train_series,
    future_covariates=cov_train_series
)

model.save("model.pt"))

predict:

model = RegressionEnsembleModel.load("model.pt"))

model.predict(
    n=args.forecast_horizon,
    series=series,
    future_covariates=future_covariates,
    num_samples=1,
    verbose=False
)

scene2：as bellow, RegressionEnsembleModel train works, but predict doesn't work, and raises an exception: AttributeError: 'NoneType' object has no attribute 'set_predict_parameters'

train:

model = RegressionEnsembleModel(
    forecasting_models = [
        TiDEModel(
            input_chunk_length=args.input_length,
            output_chunk_length=args.forecast_horizon,
            n_epochs=args.epochs,
            loss_fn=torch.nn.MSELoss(),
            model_name=model_name,
            force_reset=True,
            save_checkpoints=False,
            random_state=42,
            pl_trainer_kwargs=pl_trainer_kwargs,
            use_static_covariates=False
        ),
        NLinearModel(
            input_chunk_length=args.input_length,
            output_chunk_length=args.forecast_horizon,
            model_name=model_name,
            force_reset=True,
            save_checkpoints=False,
            random_state=42,
            pl_trainer_kwargs=pl_trainer_kwargs,
            use_static_covariates=False
        )],
    regression_train_n_points = args.input_length
)

model.fit(
    train_series,
    future_covariates=cov_train_series
)

model.save("model.pt"))

predict:

model = RegressionEnsembleModel.load("model.pt"))

model.predict(
    n=args.forecast_horizon,
    series=series,
    future_covariates=future_covariates,
    num_samples=1,
    verbose=False
)

I debugged into the code below and found that self.model is indeed None, I'm not sure if it's because of a bug?

self.model.set_predict_parameters(
    n=n,
    num_samples=num_samples,
    roll_size=roll_size,
    batch_size=batch_size,
    n_jobs=n_jobs,
    predict_likelihood_parameters=predict_likelihood_parameters,
    mc_dropout=mc_dropout,
)

To Reproduce Steps to reproduce the behavior, preferably code snippet.

Expected behavior A clear and concise description of what you expected to happen.

System (please complete the following information):

• Python version: [e.g. 3.7]
• darts version [e.g. 0.30.0]
• torch[2.0.1]
• pytorch-lightning[1.9.5]

Additional context Add any other context about the problem here.

[madtoinou]

Hi @GeorgeXiaojie,

I don't see fit() being called in the code you shared, if it isn't, the model does not exist and there is not much to actually save. Can you try the following:

from darts.models import RegressionEnsembleModel, LightGBMModel, XGBModel
from darts.datasets import AirPassengersDataset

ts = AirPassengersDataset().load()

model = RegressionEnsembleModel(
    forecasting_models = [
        LightGBMModel(
            lags=3,
            output_chunk_length=1,
            random_state=2022,
        ),
        XGBModel(
            lags=3,
            output_chunk_length=1,
            random_state=2022
        )],
    regression_train_n_points = 10
)
model.fit(ts)
# works
model.predict(3)
model.save("model.pt")

model_loaded = RegressionEnsembleModel.load("model.pt")
# works as well
model_loaded.predict(3)

[GeorgeXiaojie]

Hi @GeorgeXiaojie,

I don't see fit() being called in the code you shared, if it isn't, the model does not exist and there is not much to actually save. Can you try the following:

from darts.models import RegressionEnsembleModel, LightGBMModel, XGBModel
from darts.datasets import AirPassengersDataset

ts = AirPassengersDataset().load()

model = RegressionEnsembleModel(
  forecasting_models = [
      LightGBMModel(
          lags=3,
          output_chunk_length=1,
          random_state=2022,
      ),
      XGBModel(
          lags=3,
          output_chunk_length=1,
          random_state=2022
      )],
  regression_train_n_points = 10
)
model.fit(ts)
# works
model.predict(3)
model.save("model.pt")

model_loaded = RegressionEnsembleModel.load("model.pt")
# works as well
model_loaded.predict(3)

Thank you for your reply. There is a model.fit and I've followed up on the issue.

RegressionEnsembleModel is OK using the predictive models LightGBMModel and XGBModel.

However, RegressionEnsembleModel using TiDEModel and NLinearModel is not working.

[madtoinou]

Oh thanks for the clarification, managed to reproduce the problem.

It seems like the torch models weights are not being loaded properly, we will investigate this further.

[cnhwl]

Hi! @madtoinou @GeorgeXiaojie

I found out that the problem comes from the save and load method used by the ensemble model.

Since ensemble model uses the save of the ForecastingModel class, it is not possible to save the .ckpt file of torch model like the save method in TorchForecastingModel. In this case, even torch model does not load the .ckpt file, it's model._fit_called attribute remains to be True, which may cause AttributeError: 'NoneType' object has no attribute 'set_predict_parameters'.

We may need to figure out a way to save the .ckpt file of the torch model in the ensemble model.

[GeorgeXiaojie]

Hi! @madtoinou @GeorgeXiaojie

I found out that the problem comes from the save and load method used by the ensemble model.

Since ensemble model uses the save of the ForecastingModel class, it is not possible to save the .ckpt file of torch model like the save method in TorchForecastingModel. In this case, even torch model does not load the .ckpt file, it's model._fit_called attribute remains to be True, which may cause AttributeError: 'NoneType' object has no attribute 'set_predict_parameters'.

We may need to figure out a way to save the .ckpt file of the torch model in the ensemble model.

Thanks, looking forward to providing a solution. I've tried before to save the model directly using pickle.dump to save the model directly and then load the model using pickle.load, but I still can't get rid of the problem.