dswah
commented
5 years ago @nickcorona ahh i see this problem too! For me it occurs when the search-space is very large. What about you?
I believe this is because pygam tries to pre-determine every point in the search-space before starting the gridsearch.
The problem is that for a n-dimensional search space with m points per dimension, we can quickly get a huge array of points in n-d space: m**n.
I think correct way to create the search space would be with a generator.
Your thoughts?
The grid search feature doesn't clear memory on each iteration. This means if you're doing a very large grid search, you can run into memory issues very easily and lose all progress. Started with about 10% memory usage, and it keeps increasing with each iteration.