search

yajiemiao / pdnn

PDNN: A Python Toolkit for Deep Learning. http://www.cs.cmu.edu/~ymiao/pdnntk.html
Apache License 2.0
224 stars 105 forks source link

JSON.dump for saving nnet.tmp #7

Closed tofigh- closed 9 years ago

tofigh- commented 9 years ago

Hi yasjie, I'm wondering if there is a particular reason that you used json format to dump the nnet dictionary. As the size of the network increases, it because a huge burden since a network with four parallel conv layers may need more than 1.5G space to be saved and you also can imagine how much dump it may get to load this back to the memory.

ghost commented 9 years ago

My initial implementation used JSON (for reasons I don't remember now...). Later versions just inherited it. I am aware of this problem, but haven't figured out a better workaround. Any suggestions?

tofigh- commented 9 years ago

One solution could be writing each layer in a separate file and reading them layer by layer for initialization. But to be honest, I also don't know the optimal solution. I have to do some test with cPickle first.

tofigh- commented 9 years ago

Hi Yasjie, Besides layer-wise saving which also gives more freedom later when we only want to initialize a subset of layers, replacing json with pickle.dump also seems to help a lot to speed up the loading process.