search

yajiemiao / pdnn

PDNN: A Python Toolkit for Deep Learning. http://www.cs.cmu.edu/~ymiao/pdnntk.html
Apache License 2.0
224 stars 105 forks source link

Precision loss error #5

Closed sgrax closed 9 years ago

sgrax commented 9 years ago

I'm trying to train a net using some data, but shortly after starting training, I have this: python2 pdnn/cmds/run_DNN.py --train-data "train.pickle.gz,random=false" --valid-data "valid.pickle.gz,random=false" --nnet-spec "64:256:256:64:1" --wdir ./ --param-output-file test1.mdl --lrate "C:0.1:200" [2015-01-29 00:59:02.621588] > ... building the model [2015-01-29 00:59:02.635272] > ... getting the finetuning functions Traceback (most recent call last): File "pdnn/cmds/run_DNN.py", line 93, in batch_size=cfg.batch_size) File "/home/me/RProp/pdnn/models/dnn.py", line 162, in build_finetune_functions (index + 1) * batch_size]}) File "/usr/lib/python2.7/site-packages/theano/compile/function.py", line 223, in function profile=profile) File "/usr/lib/python2.7/site-packages/theano/compile/pfunc.py", line 512, in pfunc on_unused_input=on_unused_input) File "/usr/lib/python2.7/site-packages/theano/compile/function_module.py", line 1312, in orig_function defaults) File "/usr/lib/python2.7/site-packages/theano/compile/function_module.py", line 1192, in create defaults, self.unpack_single, self.return_none, self) File "/usr/lib/python2.7/site-packages/theano/compile/function_module.py", line 326, in init c.value = value File "/usr/lib/python2.7/site-packages/theano/gof/link.py", line 278, in set self.storage[0] = self.type.filter(value, **kwargs) File "/usr/lib/python2.7/site-packages/theano/tensor/type.py", line 152, in filter raise TypeError(err_msg, data) TypeError: ('TensorType(float32, scalar) cannot store accurately value 0.0001, it would be represented as 9.99999974738e-05. If you do not mind this precision loss, you can: 1) explicitly convert your data to a numpy array of dtype float32, or 2) set "allow_input_downcast=True" when calling "function".', 0.0001, 'Container name "learning_rate"')

The provided mnist example works.

tofigh- commented 9 years ago

I'm not the author of this package, but I may have a guess about your problem. CUDA works best with dfloat32 type variables and thus theano requires you to prepare the data type with dfloat32 as an input. My guess is, the training data you are using is dfloat64 rather than float32. To solve he problem you first need to change them to float32 by using new_data = numpy.float32(your_data)

ghost commented 9 years ago

Tofigh is right. By default, PDNN loads data into a float32 numpy array. So it's always recommended to save your data with float32.

sgrax commented 9 years ago

I was using float64. Using float32 solves the problem. Many thanks to Tofigh and Yajie!