fit model stuck in cross_validation

[DanielSunHub]

A bug occurred when I used automatic parameter training，When df is input cross_validation ，the fit process unresponsive

I can only forcibly stop the program, but I didn't find any errors，stuck in “ df_cv”row

code:

def prophetModelEvaluation(predictStartDate, df, all_params):

rmses = []

for params in all_params:
    model = Prophet(**params)
    model.add_seasonality(name='weekly', period=7, fourier_order=4, prior_scale=0.1)
    model.fit(df)
    dateTime_p = datetime.datetime.strptime(predictStartDate, '%Y-%m-%d')
    cutoffs = dateTime_p - datetime.timedelta(days=30)
    cutoffs = pd.to_datetime([cutoffs])
    df_cv = cross_validation(model, cutoffs=cutoffs, horizon='30 days', parallel="processes")
    df_p = performance_metrics(df_cv, rolling_window=1)
    rmses.append(df_p['rmse'].values[0])
    print(rmses)

best_params = all_params[np.argmin(rmses)]
print(best_params)
pass

[tcuongd]

Hmm if you set parallel=None, does it run? I can double check whether the parallelization is bugged or not, but sometimes it's dependant on the machine.

[khanetor]

I second this problem. I have been debugging, and can confirm that the problem is still there with or without parallel=None.

It happened for a particular set of parameters, i.e. given that everything stay the same, in my case, when I change the seasonality_prior_scale to 0.1, instead of 0.01 previously, then this happened. There is no error, so I don't know what is going on.

I also want to point out that this happened to me when running on AWS EC2, or Sagemaker Studio, but cross_validation was successful on my M1 Macbook Pro, given the same data and parameters.

[kadarsh22]

@DanielSunHub Can you check the value of the horizon, if the horizon is negative, this can happen.

[khanetor]

I can consistently produce this bug. Although as I mentioned, it only happened when I ran on an AWS EC2. Can I privately provide the data so that you can debug this issue @tcuongd ?