search

amplab / SparkNet

Distributed Neural Networks for Spark
MIT License
603 stars 172 forks source link

fix memory leak during broadcasting #125

Open pcmoritz opened 8 years ago

pcmoritz commented 8 years ago

found and reported by @nhe150, fixes issue #123

nhe150 commented 8 years ago

get rid of destroy since there is a lot of testing based on driver still need the variable. only put in unpersist().

Find it out the hard way.

nhe150 commented 8 years ago

there is interleaving issue to use unpersist() also.

nhe150 commented 8 years ago

find out it is due to learning rate issue. both destroy and unpersist() is good to be used. some learning rate will cause worker to learn nothing, the weight become NaN. :(

nhe150 commented 8 years ago

This is also related to https://github.com/bytedeco/javacpp-presets/issues/238