kaz-Anova / StackNet

StackNet is a computational, scalable and analytical Meta modelling framework
MIT License
1.32k stars 344 forks source link

Getting around java.lang.OutOfMemoryError #53

Closed vinkaga closed 7 years ago

vinkaga commented 7 years ago

I am trying zillow model with a different dataset. But I keep running out of memory even with -Xmx128g setting

[2.0, <=2.0]
[7.0, <=7.0]
[44142.0, <=44142.0]
 Level: 1 dimensionality: 11
 Starting cross validation
Fitting model: 1
Fitting model: 2
Fitting model: 3
Fitting model: 4
Fitting model: 5
Fitting model: 6
Fitting model: 7
Fitting model: 8
Exception in thread "Thread-1" Exception in thread "Thread-3" Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space
    at manipulate.copies.copies.Copy(copies.java:183)
    at matrix.fsmatrix.GetData(fsmatrix.java:220)
    at matrix.fsmatrix.Copy(fsmatrix.java:388)
    at ml.Bagging.BaggingRegressor.fit(BaggingRegressor.java:1443)
    at ml.Bagging.BaggingRegressor.run(BaggingRegressor.java:241)
    at java.lang.Thread.run(Thread.java:745)

I tried deleting model 8 which is GradientBoostingForestRegressor but still the same result - I guess the next model still ran out of memory.

Any suggestions?

kaz-Anova commented 7 years ago

Maybe try not to run that many models in parallel if your data set is big . So set threads to less than 8. Try with 1 and see if it runs. Running 8 models in parallel means the data set is copied 8 times.