predict() and cross_validation() outputs inconsistency

[RashidBakirov]

What happened + What you expected to happen

I am comparing the outputs of predict() and cross_validation() and it seems that they are inconsistent when using dynamic (future) features when not specifying static_features argument in cross_validation. This doesn't occur in other configurations I have tested (see below in the code).

Versions / Dependencies

python: 3.10.12 mlforecast==0.11.5 lightgbm==3.3.5 pandas==1.4.4

Reproduction script

import random
random.seed(42)

import pandas as pd
from datasetsforecast.m4 import M4
from utilsforecast.plotting import plot_series
import numpy as np

import lightgbm as lgb
from mlforecast import MLForecast
from mlforecast.target_transforms import Differences
from numba import njit
from window_ops.expanding import expanding_mean
from window_ops.rolling import rolling_mean

def hour_index(times):
    return times % 24

def init_fcst():
    fcst = MLForecast(
    models=lgb.LGBMRegressor(**lgb_params),
    freq=1,
    target_transforms=[Differences([24])],
    lags=[1, 24],
    lag_transforms={
        1: [expanding_mean],
        24: [(rolling_mean, 48)],
    },
    date_features=[hour_index],
    )

    return fcst

lgb_params = {
    'verbosity': -1,
    'num_leaves': 512,
}

await M4.async_download('data', group='Hourly')
df, *_ = M4.load('data', 'Hourly')
uids = df['unique_id'].unique()
random.seed(0)
sample_uids = random.choices(uids, k=4)
df = df[df['unique_id'].isin(sample_uids)].reset_index(drop=True)
df['ds'] = df['ds'].astype('int64')
df['feat']=[random.randint(-2,2)  for k in df.index]

print(df.head())
print()
print('No features')
print('--------------------------')

print('Predict output')
fcst=init_fcst()
fcst.fit(df.loc[df.ds<1008,['unique_id','ds','y']])
print(fcst.predict(1))

print('CV output')
fcst=init_fcst()
print(fcst.cross_validation(df[['unique_id','ds','y']],n_windows=1,h=1))

print()
print('With static features')
print('--------------------------')

print('Predict output specifying static features')
fcst=init_fcst()
fcst.fit(df[df.ds<1008],static_features=['feat'])
print(fcst.predict(1))

print('Predict output not specifying static features')
fcst=init_fcst()
fcst.fit(df[df.ds<1008])
print(fcst.predict(1))

print('CV output')
fcst=init_fcst()
print(fcst.cross_validation(df,n_windows=1,h=1,static_features=['feat']))

print()
print('With dynamic features')
print('--------------------------')

print('Predict output')
fcst=init_fcst()
fcst.fit(df[df.ds<1008],static_features=[])
print(fcst.predict(1, X_df=df[['unique_id','ds','feat']]))

print('CV output specifying static_features')
fcst=init_fcst()
print(fcst.cross_validation(df,n_windows=1,h=1,static_features=[]))

print('CV output not specifying static_features')
fcst=init_fcst()
print(fcst.cross_validation(df,n_windows=1,h=1))

Output:

  unique_id  ds     y  feat
0      H196   1  11.8     2
1      H196   2  11.4     1
2      H196   3  11.1     1
3      H196   4  10.8     0
4      H196   5  10.6     1

No features
--------------------------
Predict output
  unique_id    ds  LGBMRegressor
0      H196  1008      17.023168
1      H256  1008      13.401523
2      H381  1008     228.326448
3      H413  1008      36.263064
CV output
  unique_id    ds  cutoff      y  LGBMRegressor
0      H196  1008    1007   16.8      17.023168
1      H256  1008    1007   13.4      13.401523
2      H381  1008    1007  207.0     228.326448
3      H413  1008    1007   34.0      36.263064

With static features
--------------------------
Predict output specifying static features
  unique_id    ds  LGBMRegressor
0      H196  1008      16.818113
1      H256  1008      13.434333
2      H381  1008     220.357912
3      H413  1008      38.192499
Predict output not specifying static features
  unique_id    ds  LGBMRegressor
0      H196  1008      16.818113
1      H256  1008      13.434333
2      H381  1008     220.357912
3      H413  1008      38.192499
CV output
  unique_id    ds  cutoff      y  LGBMRegressor
0      H196  1008    1007   16.8      16.818113
1      H256  1008    1007   13.4      13.434333
2      H381  1008    1007  207.0     220.357912
3      H413  1008    1007   34.0      38.192499

With dynamic features
--------------------------
Predict output
  unique_id    ds  LGBMRegressor
0      H196  1008      16.940767
1      H256  1008      13.334591
2      H381  1008     220.357912
3      H413  1008      37.280410
CV output specifying static_features
  unique_id    ds  cutoff      y  LGBMRegressor
0      H196  1008    1007   16.8      **16.940767**
1      H256  1008    1007   13.4      **13.334591**
2      H381  1008    1007  207.0     220.357912
3      H413  1008    1007   34.0      **37.280410**
CV output not specifying static_features
  unique_id    ds  cutoff      y  LGBMRegressor
0      H196  1008    1007   16.8      **16.818113**
1      H256  1008    1007   13.4      **13.434333**
2      H381  1008    1007  207.0     220.357912
3      H413  1008    1007   34.0      **38.192499**

Issue Severity

Low: It annoys or frustrates me.

[jmoralez]

Hey @RashidBakirov, thanks for using mlforecast. If you don't specify static_features they're all interpreted as static, so setting static_features=None (the default) or static_features=['feat'] is the same. I believe the outputs are correct here, aren't they? In one of the cases feat is being intepreted as static and in the other case as dynamic.

[RashidBakirov]

Hi @jmoralez

According to the documentation of cross_validation, the final two cross_validation calls fcst.cross_validation(df,n_windows=1,h=1,static_features=[]) and fcst.cross_validation(df,n_windows=1,h=1) should have resulted in the same outcome, with feat being treated as dynamic, at least this is my interpretation. This was also an impression I got from my previous question on this https://github.com/Nixtla/mlforecast/discussions/144 . However, it looks omitting static_features argument assigns feat to be static.

Perhaps the documentation should be made more explicit to avoid misunderstandings.

[jmoralez]

As I said in my previous answer, the default is to treat all features as static, so:

• static_features=None (the default) -> static = ['feat'], dynamic = []
• static_features=['feat'] -> static = ['feat'], dynamic = []
• static_features=[] -> static = [], dynamic = ['feat']

So it's expected for those two commands to differ, since in the first one feat is dynamic and in the second one feat is static. That's the same case with predict, so if you don't set static_features and then provide X_df you'll get an error. It'd be expensive to validate if the static features are actually static (constant) but we could add something to see if the first and last rows don't match and raise an error.

[RashidBakirov]

As I said in my previous answer, the default is to treat all features as static, so:

• static_features=None (the default) -> static = ['feat'], dynamic = []

I understand this now, but I do feel that the above is confusing, and at least could be explicitly mentioned in the documentation.

It'd be expensive to validate if the static features are actually static (constant) but we could add something to see if the first and last rows don't match and raise an error.

I would rather keep the current (replication of the last value) behaviour for static features even if they are not quite static, but their future values are not known.

Thanks for your help!

[jmoralez]

Would you like to modify the wording here a bit to make it clearer?

[github-actions[bot]]

This issue has been automatically closed because it has been awaiting a response for too long. When you have time to to work with the maintainers to resolve this issue, please post a new comment and it will be re-opened. If the issue has been locked for editing by the time you return to it, please open a new issue and reference this one.