search

facebook / prophet

Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.
https://facebook.github.io/prophet
MIT License
18.55k stars 4.54k forks source link

Prophet search trial freezes with optuna #2357

Open dariusmargineansrx opened 1 year ago

dariusmargineansrx commented 1 year ago

Hello!

I am currently using Prophet for a forecasting prototype, and I've done the search for the hyperparameters using optuna TPESampler. I have around 300 dataframes on which I have to compute the hyperparameter optimizations for. After completing a couple of dataframes the search suddenly freezes without any warning message and without exiting the proccess. It just gets blocked at a certain nr of trial (for the current search it has stopped, after completing 130 dataframes hyperparameter optimization at trial 786 out of 1000). I've tried to increase and decrease the number of trials per dataframe & (as expected) I've seen that as long as the number of trials goes down, more dataframes gets to be computed & the more the number of trials goes up (like 2000) less dataframes gets to be computed. Has everyone encountered this before? It really gets annoying because I have to restart the search because it can never get to an end. I am not quite sure if this is a problem with optuna or Prophet so I will post this on their github issue too.

It gets stucked inside the search function I've created, in the cross_validation() method from Prophet (I can tell that because in here it creates parallel processes = how many folds we want to test the subset of hyperparameters, in my code are defined as cutoffs)

Here is the code I am using:

def generate_cutoffs(df, weeks_to_forecast):
    cutoffs = pd.to_datetime(
            [
                df.at[len(df) - (weeks_to_forecast*9+1), "ds"],
                df.at[len(df) - (weeks_to_forecast*8+1), "ds"],
                df.at[len(df) - (weeks_to_forecast*7+1), "ds"],
                df.at[len(df) - (weeks_to_forecast*6+1), "ds"],
                df.at[len(df) - (weeks_to_forecast*5+1), "ds"],
                df.at[len(df) - (weeks_to_forecast*4+1), "ds"],
                df.at[len(df) - (weeks_to_forecast*3+1), "ds"],
                df.at[len(df) - (weeks_to_forecast*2+1), "ds"],
                df.at[len(df) - (weeks_to_forecast+1), "ds"],
            ]
        )
    return cutoffs

def get_best_hp_prophet_cv(df_for_cv, weeks_to_forecast, holidays_df):
    logging.getLogger('prophet').setLevel(logging.ERROR)
    logging.getLogger('fbprophet').setLevel(logging.ERROR)

    # print('DF entered for search: ', df_for_cv)
    cutoffs = generate_cutoffs(df_for_cv, weeks_to_forecast)

    def objective(trial):

        # print(cutoffs)

        # print(df.tail(30))
        param_grid = {
        "changepoint_prior_scale": trial.suggest_categorical(
            "changepoint_prior_scale", [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0]
        ),
        "seasonality_prior_scale": trial.suggest_categorical(
            "seasonality_prior_scale", [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0]

        ),
        "seasonality_mode": trial.suggest_categorical(
            "seasonality_mode", ["multiplicative", "additive"]
        ),
        "growth": trial.suggest_categorical("growth", ["linear"]),
        "yearly_seasonality": trial.suggest_categorical(
            "yearly_seasonality", [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17]
        ),
        "daily_seasonality": trial.suggest_categorical("daily_seasonality",[False]),
        "weekly_seasonality": trial.suggest_categorical("weekly_seasonality",[False]),
        "uncertainty_samples": trial.suggest_categorical("uncertainty_samples",[0]),
    }

        prior_scale_month = trial.suggest_categorical('prior_scale_month', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_week_num = trial.suggest_categorical('prior_scale_week_num', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_month_qty_over_df = trial.suggest_categorical('prior_scale_avg_month_qty_over_df', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_week_nr_qty_over_df = trial.suggest_categorical('prior_scale_avg_week_nr_qty_over_df', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])        

        prior_scale_avg_solar_rad_of_month = trial.suggest_categorical('prior_scale_avg_solar_rad_of_month', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_solar_rad_of_week_nr = trial.suggest_categorical('prior_scale_avg_solar_rad_of_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_temp_of_month = trial.suggest_categorical('prior_scale_avg_temp_of_month', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_temp_of_week_nr = trial.suggest_categorical('prior_scale_avg_temp_of_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_pres_of_month = trial.suggest_categorical('prior_scale_avg_pres_of_month', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_pres_of_week_nr = trial.suggest_categorical('prior_scale_avg_pres_of_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])

        prior_scale_avg_solar_rad_last_week_week_nr = trial.suggest_categorical('prior_scale_avg_solar_rad_last_week_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_temp_last_week_week_nr = trial.suggest_categorical('prior_scale_avg_temp_last_week_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_pres_last_week_week_nr = trial.suggest_categorical('prior_scale_avg_pres_last_week_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])

        prior_scale_avg_solar_rad_last_2weeks_week_nr = trial.suggest_categorical('prior_scale_avg_solar_rad_last_2weeks_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_temp_last_2weeks_week_nr = trial.suggest_categorical('prior_scale_avg_temp_last_2weeks_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_pres_last_2weeks_week_nr = trial.suggest_categorical('prior_scale_avg_pres_last_2weeks_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])

        prior_scale_avg_solar_rad_next_week_week_nr = trial.suggest_categorical('prior_scale_avg_solar_rad_next_week_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_temp_next_week_week_nr = trial.suggest_categorical('prior_scale_avg_temp_next_week_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_pres_next_week_week_nr = trial.suggest_categorical('prior_scale_avg_pres_next_week_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])

        prior_scale_avg_solar_rad_of_last_month = trial.suggest_categorical('prior_scale_avg_solar_rad_of_last_month', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_temp_of_last_month = trial.suggest_categorical('prior_scale_avg_temp_of_last_month', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_pres_of_last_month = trial.suggest_categorical('prior_scale_avg_pres_of_last_month', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])

        prior_scale_avg_qty_last_2weeks_week_nr = trial.suggest_categorical('prior_scale_avg_qty_last_2weeks_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])
        prior_scale_avg_qty_next_week_week_nr = trial.suggest_categorical('prior_scale_avg_qty_next_week_week_nr', [0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.09, 0.1, 0.2, 0.5, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 5.0, 9.0, 10.0])

        # I ve used this only for testing to see if everything works fine
        # param_grid = {
        # 'changepoint_prior_scale': trial.suggest_categorical('changepoint_prior_scale', [0.001]),
        # 'seasonality_prior_scale': trial.suggest_categorical('seasonality_prior_scale',[0.01, 0.1]),
        # 'seasonality_mode' : trial.suggest_categorical('seasonality_mode',['additive']),
        # 'growth': trial.suggest_categorical('growth',['linear']),
        # 'yearly_seasonality': trial.suggest_categorical('yearly_seasonality',[14,15]),
        # 'holidays_prior_scale' : trial.suggest_categorical('holidays_prior_scale',[10])
        # }

        # all_params = [dict(zip(param_grid.keys(), v)) for v in itertools.product(*param_grid.values())]
        # mses = []  # Store the MSEs for each params here

        # Use cross validation to evaluate all parameters
        # for params in all_params:
        m = Prophet(**param_grid, holidays = holidays_df)
        m.add_regressor('month', prior_scale = prior_scale_month)
        m.add_regressor('week_num', prior_scale = prior_scale_week_num)
        m.add_regressor('avg_month_qty_over_df', prior_scale = prior_scale_avg_month_qty_over_df)
        m.add_regressor('avg_week_nr_qty_over_df', prior_scale = prior_scale_avg_week_nr_qty_over_df)

        m.add_regressor('avg_solar_rad_of_month', prior_scale = prior_scale_avg_solar_rad_of_month)
        m.add_regressor('avg_solar_rad_of_week_nr', prior_scale = prior_scale_avg_solar_rad_of_week_nr)
        m.add_regressor('avg_pres_of_month', prior_scale = prior_scale_avg_pres_of_month)
        m.add_regressor('avg_pres_of_week_nr', prior_scale = prior_scale_avg_pres_of_week_nr)
        m.add_regressor('avg_temp_of_month', prior_scale = prior_scale_avg_temp_of_month)
        m.add_regressor('avg_temp_of_week_nr', prior_scale = prior_scale_avg_temp_of_week_nr)

        m.add_regressor('avg_solar_rad_last_week_week_nr', prior_scale = prior_scale_avg_solar_rad_last_week_week_nr)
        m.add_regressor('avg_temp_last_week_week_nr', prior_scale = prior_scale_avg_temp_last_week_week_nr)
        m.add_regressor('avg_pres_last_week_week_nr', prior_scale = prior_scale_avg_pres_last_week_week_nr)

        m.add_regressor('avg_solar_rad_last_2weeks_week_nr', prior_scale = prior_scale_avg_solar_rad_last_2weeks_week_nr)
        m.add_regressor('avg_temp_last_2weeks_week_nr', prior_scale = prior_scale_avg_temp_last_2weeks_week_nr)
        m.add_regressor('avg_pres_last_2weeks_week_nr', prior_scale = prior_scale_avg_pres_last_2weeks_week_nr)

        m.add_regressor('avg_solar_rad_next_week_week_nr', prior_scale = prior_scale_avg_solar_rad_next_week_week_nr)
        m.add_regressor('avg_temp_next_week_week_nr', prior_scale = prior_scale_avg_temp_next_week_week_nr)
        m.add_regressor('avg_pres_next_week_week_nr', prior_scale = prior_scale_avg_pres_next_week_week_nr)

        m.add_regressor('avg_solar_rad_of_last_month', prior_scale = prior_scale_avg_solar_rad_of_last_month)
        m.add_regressor('avg_temp_of_last_month', prior_scale = prior_scale_avg_temp_of_last_month)
        m.add_regressor('avg_pres_of_last_month', prior_scale = prior_scale_avg_pres_of_last_month)

        m.add_regressor('avg_qty_last_2weeks_week_nr', prior_scale = prior_scale_avg_qty_last_2weeks_week_nr)
        m.add_regressor('avg_qty_next_week_week_nr', prior_scale = prior_scale_avg_qty_next_week_week_nr)

        print("Model_params currently are: ", param_grid)
        dict_with_non_model_params_to_print = {
            'month': prior_scale_month,
            'week_num': prior_scale_week_num,
            'avg_month_qty_over_df': prior_scale_avg_month_qty_over_df,
            'avg_week_nr_qty_over_df': prior_scale_avg_week_nr_qty_over_df,

            'avg_solar_rad_of_month': prior_scale_avg_solar_rad_of_month,
            'avg_solar_rad_of_week_nr': prior_scale_avg_solar_rad_of_week_nr,
            'avg_pres_of_month': prior_scale_avg_pres_of_month,
            'avg_pres_of_week_nr': prior_scale_avg_pres_of_week_nr,
            'avg_temp_of_month': prior_scale_avg_temp_of_month,
            'avg_temp_of_week_nr': prior_scale_avg_temp_of_week_nr,

            'avg_solar_rad_last_week_week_nr': prior_scale_avg_solar_rad_last_week_week_nr,
            'avg_temp_last_week_week_nr': prior_scale_avg_temp_last_week_week_nr,
            'avg_pres_last_week_week_nr': prior_scale_avg_pres_last_week_week_nr,

            'avg_solar_rad_last_2weeks_week_nr':prior_scale_avg_solar_rad_last_2weeks_week_nr,
            'avg_temp_last_2weeks_week_nr': prior_scale_avg_temp_last_2weeks_week_nr,
            'avg_pres_last_2weeks_week_nr': prior_scale_avg_pres_last_2weeks_week_nr,

            'avg_solar_rad_next_week_week_nr': prior_scale_avg_solar_rad_next_week_week_nr,
            'avg_temp_next_week_week_nr': prior_scale_avg_temp_next_week_week_nr,
            'avg_pres_next_week_week_nr': prior_scale_avg_pres_next_week_week_nr,

            'avg_solar_rad_of_last_month': prior_scale_avg_solar_rad_of_last_month,
            'avg_temp_of_last_month': prior_scale_avg_temp_of_last_month,
            'avg_pres_of_last_month': prior_scale_avg_pres_of_last_month,

            'avg_qty_last_2weeks_week_nr': prior_scale_avg_qty_last_2weeks_week_nr,
            'avg_qty_next_week_week_nr': prior_scale_avg_qty_next_week_week_nr
        }
        print("Non_model_params_currently are: ", dict_with_non_model_params_to_print)

        # m.add_country_holidays(country_name='UK')
        # m.add_regressor('month')
        # m.add_regressor('week_num')
        # m.add_regressor('avg_month_qty_over_df')
        # m.add_regressor('avg_week_nr_qty_over_df')
        # m.add_regressor('BH_Minus_1_Week')
        # m.add_regressor('BH_Minus_4_Weekday')
        # m.add_regressor('BH_Minus_3_Weekday')
        # m.add_regressor('BH_Minus_2_Weekday')
        # m.add_regressor('BH_Minus_1_Weekday')
        # m.add_regressor('BH_Nearest_Sat')
        # m.add_regressor('Between_Xmas_NY')
        # m.add_regressor('Xmas_NY')
        # m.add_regressor('BH')
        # m.add_regressor('BH_Plus_1_Weekday')
        # m.add_regressor('BH_Plus_2_Weekday')
        # m.add_regressor('BH_Plus_3_Weekday')
        # m.add_regressor('BH_Plus_4_Weekday')
        # m.add_regressor('BH_Plus_1_Week')
        # m.add_regressor('Holiday_Xmas_Before_After')
        # m.add_regressor('Holiday_Xmas')
        # m.add_regressor('Holiday_Easter_Before_After')
        # m.add_regressor('Holiday_Easter')
        # m.add_regressor('Holiday_Summer_Before_After')
        # m.add_regressor('Holiday_Summer')
        # m.add_regressor('Half_Term')

        m.fit(df_for_cv)
        df_cv = cross_validation(
            m, cutoffs=cutoffs, horizon="{} days".format(weeks_to_forecast*7), parallel="processes"
        )
        df_p = performance_metrics(df_cv, rolling_window=1)
        return df_p["mse"].values[0]

    # Find the best parameters
    optuna_prophet = optuna.create_study(
        direction="minimize", sampler=TPESampler(seed=321)
    )

    # * n_trials optuna hyperparameter.
    # optuna_prophet.optimize(objective, n_trials=5000)
    optuna_prophet.optimize(objective, n_trials=1000)

    prophet_trial = optuna_prophet.best_trial
    prophet_trial_params = prophet_trial.params

    list_of_variables_outside_the_param_grid = ['prior_scale_month',
                                                'prior_scale_week_num',
                                                'prior_scale_avg_month_qty_over_df',
                                                'prior_scale_avg_week_nr_qty_over_df', 

                                                'prior_scale_avg_solar_rad_of_month', 
                                                'prior_scale_avg_solar_rad_of_week_nr', 
                                                'prior_scale_avg_temp_of_month', 
                                                'prior_scale_avg_temp_of_week_nr', 
                                                'prior_scale_avg_pres_of_month', 
                                                'prior_scale_avg_pres_of_week_nr',

                                                'prior_scale_avg_solar_rad_last_week_week_nr',
                                                'prior_scale_avg_temp_last_week_week_nr', 
                                                'prior_scale_avg_pres_last_week_week_nr',

                                                'prior_scale_avg_solar_rad_last_2weeks_week_nr',
                                                'prior_scale_avg_temp_last_2weeks_week_nr',
                                                'prior_scale_avg_pres_last_2weeks_week_nr',

                                                'prior_scale_avg_solar_rad_next_week_week_nr',
                                                'prior_scale_avg_temp_next_week_week_nr',
                                                'prior_scale_avg_pres_next_week_week_nr',

                                                'prior_scale_avg_solar_rad_of_last_month',
                                                'prior_scale_avg_temp_of_last_month',
                                                'prior_scale_avg_pres_of_last_month',

                                                'prior_scale_avg_qty_last_2weeks_week_nr',
                                                'prior_scale_avg_qty_next_week_week_nr']

    params_outside_the_param_grid={}
    param_grid = {}
    for param_name in prophet_trial_params.keys():
        if param_name in list_of_variables_outside_the_param_grid:
            params_outside_the_param_grid.update({param_name : prophet_trial_params[param_name]})
        else:
            param_grid.update({param_name : prophet_trial_params[param_name]})

    return param_grid, params_outside_the_param_grid

The 300 dataframes on which I tried to optimize the Prophet model for, are about >280 rows & < 290 rows each. I do a 9 fold cross_validation using cuttofs (a list datetime objects, they are created using the generate_cutoffs() where df is the dataframe I want to compute the hyperparameter optimization, and weeks_to_forecast is always 12)

Has anybody experienced something similar? Or does anybody know any workarounds?

tcuongd commented 1 year ago

Thanks for reporting, my initial guess is that there's an issue with cmdstanpy creating temp files (to hold optimisation results) and then not cleaning them up, which fails for really large searches.

This issue might be related: https://github.com/facebook/prophet/issues/2355

I'll look into this for the next release

rdunlap-inspire11 commented 9 months ago

We've been monitoring this ticket and the following similar tickets since March 2023, since we also have had occasional hanging task issues. We've frozen on prophet 1.1 until we feel we can trust a newer version of prophet to not freeze; any progress towards resolution?