20% performance regression in 1.1.2/1.1.3

[nicolaerosia]

Hello,

I believe there is a performance regression between 1.1.1 and 1.1.2

Simple reproducer

import statistics
import timeit

import pandas as pd
import prophet

# https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_peyton_manning.csv
#df_in = pd.read_csv("./example_wp_log_peyton_manning.csv")
df_in = pd.read_csv("https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_peyton_manning.csv")

def run_prophet_speedtest(periods=30, include_history=False, holiday=True):
    m = prophet.Prophet(mcmc_samples=0, uncertainty_samples=1000)
    if holiday:
        m.add_country_holidays(country_name="US")
    m.fit(df_in)

    # Python
    future = m.make_future_dataframe(periods=periods, include_history=include_history)
    forecast = m.predict(future)
    return m, forecast

t = timeit.Timer(lambda: run_prophet_speedtest())

ret = t.repeat(repeat=10, number=6)
mean = statistics.mean(ret)
median = statistics.median(ret)
std = statistics.stdev(ret)

print(f"results: {ret}")
print(f"mean: {mean}, median: {median}, std: {std}")

try with:

pip install prophet==1.1.1
pip install prophet==1.1.2

I have also my own build on which I have cherry picked some commits, and I get a speed increase compared to 1.1.1, with this commit: https://github.com/facebook/prophet/commit/07167512883f2035c8f3b2dada2a64f2cf698bdf

I still need to bisect, but I suspect it's either the fourier related commits or this warm start, https://github.com/facebook/prophet/commit/eb71d0117b7154e2b584e11fd2b7ea7aa13a2128 . Any ideas?

[tcuongd]

Thanks heaps! Is it possible to time each of the fit and predict steps separately to identify the slow part?

The warm starts code path should only be run when init is passed, which it wasn't in your example.

[nicolaerosia]

@tcuongd

it's even worse, here is a quick snippet you can try. Testing on Python 3.9 because 1.1.1 was not supporting 3.11.

rm -rf venv-1
python3.9 -m venv venv-1
source venv-1/bin/activate

pip install prophet==1.1.1 holidays==0.24
pip freeze -> venv-1-pip-freeze.txt

python3 prophet_benchmark.py

Results:

fit only
results: [2.6406618790006178, 2.5883878200002073, 2.618681915000707, 2.597276200000124, 2.590181763998771, 2.6008756900009757, 2.6040572540005087, 2.608617838999635, 2.789804326999729, 2.65691634399991]
mean: 2.6295461032001186, median: 2.6063375465000718, std: 0.060431768532746344
fit + predict
results: [2.714795433999825, 2.850421868999547, 2.638496886000212, 2.6244312430007994, 2.631157390000226, 2.6112251470003685, 2.6814545750003163, 2.980850472000384, 2.610891910000646, 2.7252798459994665]
mean: 2.706900477200179, median: 2.659975730500264, std: 0.12123589033861379

rm -rf venv-3
python3.9 -m venv venv-3
source venv-3/bin/activate
pip install prophet==1.1.3

pip freeze -> venv-3-pip-freeze.txt

python3 prophet_benchmark.py

Results:

only fit
results: [3.2665141450015653, 3.206068987001345, 3.2111546550004277, 3.2630801260002045, 3.067265342999235, 3.2855859010014683, 3.109610724999584, 3.112001972998769, 3.364600740000242, 3.5069349340010376]
mean: 3.239281752900388, median: 3.237117390500316, std: 0.13135773814410753
fit + predict
results: [3.0524819580004987, 3.670144108000386, 3.2175543370012747, 3.236598290000984, 3.2881391849987267, 3.197264123000423, 3.1189955129993905, 3.273076322000634, 3.317247042001327, 3.2037827340009244]
mean: 3.257528361200457, median: 3.2270763135011293, std: 0.16492037667999565

Test script

import statistics
import timeit

import pandas as pd
import prophet

# https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_peyton_manning.csv
# df_in = pd.read_csv("./example_wp_log_peyton_manning.csv")
df_in = pd.read_csv(
    "https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_peyton_manning.csv"
)

def run_prophet_speedtest_only_fit(periods=30, include_history=False, holiday=True):
    m = prophet.Prophet(mcmc_samples=0, uncertainty_samples=1000)
    if holiday:
        m.add_country_holidays(country_name="US")
    m.fit(df_in)

    # Python
    future = m.make_future_dataframe(periods=periods, include_history=include_history)
    forecast = m.predict(future)
    return m, forecast

def run_prophet_speedtest(periods=30, include_history=False, holiday=True):
    m = prophet.Prophet(mcmc_samples=0, uncertainty_samples=1000)
    if holiday:
        m.add_country_holidays(country_name="US")
    m.fit(df_in)

    # Python
    future = m.make_future_dataframe(periods=periods, include_history=include_history)
    forecast = m.predict(future)
    return m, forecast

t = timeit.Timer(lambda: run_prophet_speedtest_only_fit())

ret = t.repeat(repeat=10, number=6)
mean = statistics.mean(ret)
median = statistics.median(ret)
std = statistics.stdev(ret)

print("only fit")
print(f"results: {ret}")
print(f"mean: {mean}, median: {median}, std: {std}")

t = timeit.Timer(lambda: run_prophet_speedtest())

ret = t.repeat(repeat=10, number=6)
mean = statistics.mean(ret)
median = statistics.median(ret)
std = statistics.stdev(ret)

print("fit + predict")
print(f"results: {ret}")
print(f"mean: {mean}, median: {median}, std: {std}")

[nicolaerosia]

@tcuongd hang on, seems to be fixed in 1.1.4, redoing my tests

[tcuongd]

Oh interesting. What platform are you on btw?