search

facebook / prophet

Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.
https://facebook.github.io/prophet
MIT License
18.28k stars 4.51k forks source link

Parallel cross validation & memory overflow #2018

Open kboghe opened 3 years ago

kboghe commented 3 years ago

Hi,

I'm trying to tune six forecasts using a parameter grid (with a total of ~100 combinations) in a straightforward for loop. I'm using the parallel processing option while cross validating, but for some reason the loop takes up more and more memory as it progresses through the parameter grid, eventually leading to memory overflow. This seems odd to me since all variables are being replaced after each sequantial run. I suspect this has something to do with how parallel processes are (not) being terminated and thus eating up more and more of the server's RAM? It doesn't matter whether I use the 'processes' or 'threads' option.

I have no idea how this parallelization works on the backend; is there some kind of progress I need to terminate after each run?

Thanks!

nviet commented 3 years ago

I'm also running cross validation (usually 4 or 5 at the same time) with a parameter grid of >400 combinations in a straightforward loop too. It runs just fine and consumes only about 6-7 GB of RAM I think, but CPU usage is always near 100%. Maybe you can post some sample code so that other people can try on their system?