search

dwright04 / ufldl

Reimplementations of unsupervised feature learning and deep learning algorithms in python
0 stars 1 forks source link

Multiprocessing appears to keep multiple copies of data. #4

Open dwright04 opened 8 years ago

dwright04 commented 8 years ago

Multiprocessing appears to keep multiple copies of the data in memory one for the master and another shared by the spawned processes.

Ken’s suggestion is to spawn process to load data which is returned to master and then passed to subsequent spawned processes. I’m not sure how this can work as even if the process which loads the data dies, the master still has a copy of the data...

dwright04 commented 7 years ago

Memory is still a problem for the convolutional and pooling stage.