search

uoguelph-mlrg / Theano-MPI

MPI Parallel framework for training deep learning models built in Theano
Other
53 stars 21 forks source link

cnmem wrong result #8

Open FredericMao opened 8 years ago

FredericMao commented 8 years ago

Hi He,

The wrong result when using cnmem is caused by para-loading. I tested with code in /home/mahe6562/Documents/lasagne_googlenet/4-4/alex-stale2-8gpu-cop4_adjust_mkl, if turned off para-load, it gave correct result. If turned on para-load, it showed NAN. I tested both one GPU and two GPU.

It is not a hurry to fix this problem.

hma02 commented 8 years ago

Okay. I wil take a look at this when I have time.