Error when training a model

[antxa]

Hi! I'm trying to train my model but I get this error:

./train.py --proto=prototype_encdec_state --state mydata.py

python-2.7/lib/python2.7/site-packages/theano/sandbox/rng_mrg.py:768: UserWarning: MRG_RandomStreams Can't determine #streams from size (Elemwise{Cast{int32}}.0), guessing 60*256
  nstreams = self.n_streams(size)
Traceback (most recent call last):
  File "./train.py", line 100, in <module>
    main()
  File "./train.py", line 81, in main
    enc_dec.build()
  File "build/bdist.linux-x86_64/egg/experiments/nmt/encdec.py", line 1385, in build
    c=self.sampling_c)
  File "build/bdist.linux-x86_64/egg/experiments/nmt/encdec.py", line 1252, in build_sampler
    name="{}_sampler_scan".format(self.prefix))
  File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/theano/scan_module/scan.py", line 1007, in scan
    scan_outs = local_op(*scan_inputs)
  File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/theano/gof/op.py", line 399, in __call__
    node = self.make_node(*inputs, **kwargs)
  File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 370, in make_node
    inner_sitsot_out.type.ndim))
ValueError: When compiling the inner function of scan the following error has been encountered: The initial state (outputs_info in scan nomenclature) of variable IncSubtensor{Set;:int64:}.0 (argument number 2) has dtype float32 and 2 dimension(s), while the result of the inner function for this output has dtype float64 and 1 dimension(s). This could happen if the inner graph of scan results in an upcast or downcast. Please make sure that you use dtypes consistently

I'm new on neural networks and GroundHog and I don't understand the error message. Could anyone help me with it?

Thanks!

[nouiz]

Update theano. It is probably fixed. Be site to use the development version of theano or the release candidate for 0.7 Le 18 mars 2015 09:29, "antxa" notifications@github.com a écrit :

Hi! I'm trying to train my model but I get this error:

./train.py --proto=prototype_encdec_state --state mydata.py

python-2.7/lib/python2.7/site-packages/theano/sandbox/rng_mrg.py:768: UserWarning: MRG_RandomStreams Can't determine #streams from size (Elemwise{Cast{int32}}.0), guessing 60_256 nstreams = self.n_streams(size) Traceback (most recent call last): File "./train.py", line 100, in main() File "./train.py", line 81, in main enc_dec.build() File "build/bdist.linux-x86_64/egg/experiments/nmt/encdec.py", line 1385, in build c=self.sampling_c) File "build/bdist.linux-x86_64/egg/experiments/nmt/encdec.py", line 1252, in build_sampler name="{}_sampler_scan".format(self.prefix)) File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/theano/scan_module/scan.py", line 1007, in scan scan_outs = local_op(_scan_inputs) File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/theano/gof/op.py", line 399, in call node = self.make_node(_inputs, *_kwargs) File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 370, in make_node inner_sitsot_out.type.ndim)) ValueError: When compiling the inner function of scan the following error has been encountered: The initial state (outputs_info in scan nomenclature) of variable IncSubtensor{Set;:int64:}.0 (argument number 2) has dtype float32 and 2 dimension(s), while the result of the inner function for this output has dtype float64 and 1 dimension(s). This could happen if the inner graph of scan results in an upcast or downcast. Please make sure that you use dtypes consistently

I'm new on neural networks and GroundHog and I don't understand the error message. Could anyone help me with it?

Thanks!

— Reply to this email directly or view it on GitHub https://github.com/lisa-groundhog/GroundHog/issues/28.

[antxa]

I've updated Theano, but I get the same error:

Traceback (most recent call last):
  File "./train.py", line 100, in <module>
    main()
  File "./train.py", line 81, in main
    enc_dec.build()
  File "build/bdist.linux-x86_64/egg/experiments/nmt/encdec.py", line 1385, in build
    c=self.sampling_c)
  File "build/bdist.linux-x86_64/egg/experiments/nmt/encdec.py", line 1252, in build_sampler
    name="{}_sampler_scan".format(self.prefix))
  File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/Theano-0.7.0rc2-py2.7.egg/theano/scan_module/scan.py", line 1017, in scan
    scan_outs = local_op(*scan_inputs)
  File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/Theano-0.7.0rc2-py2.7.egg/theano/gof/op.py", line 507, in __call__
    node = self.make_node(*inputs, **kwargs)
  File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/Theano-0.7.0rc2-py2.7.egg/theano/scan_module/scan_op.py", line 350, in make_node
    inner_sitsot_out.type.dtype))
ValueError: When compiling the inner function of scan the following error has been encountered: The initial state (`outputs_info` in scan nomenclature) of variable IncSubtensor{Set;:int64:}.0 (argument number 2) has dtype float32, while the result of the inner function (`fn`) has dtype float64. This can happen if the inner function of scan results in an upcast or downcast.

[nouiz]

Do you have the Theano flag floatX to float32 or float64? The problem is that some initial value of the scan step are in float32, but the new value have dtype float64.

You can use the theano flag warn_float64=pdb to get in the python debogger when the upcast happen.

On Mon, Mar 23, 2015 at 6:56 AM, antxa notifications@github.com wrote:

I've updated Theano, but I get the same error:

Traceback (most recent call last): File "./train.py", line 100, in main() File "./train.py", line 81, in main enc_dec.build() File "build/bdist.linux-x86_64/egg/experiments/nmt/encdec.py", line 1385, in build c=self.sampling_c) File "build/bdist.linux-x86_64/egg/experiments/nmt/encdec.py", line 1252, in build_sampler name="{}_sampler_scan".format(self.prefix)) File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/Theano-0.7.0rc2-py2.7.egg/theano/scan_module/scan.py", line 1017, in scan scan_outs = local_op(scan_inputs) File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/Theano-0.7.0rc2-py2.7.egg/theano/gof/op.py", line 507, in call node = self.make_node(_inputs, _kwargs) File "/soft_orokorra_linux_x86_64_rhel6/python-2.7/lib/python2.7/site-packages/Theano-0.7.0rc2-py2.7.egg/theano/scan_module/scan_op.py", line 350, in make_node inner_sitsot_out.type.dtype)) ValueError: When compiling the inner function of scan the following error has been encountered: The initial state (outputs_info in scan nomenclature) of variable IncSubtensor{Set;:int64:}.0 (argument number 2) has dtype float32, while the result of the inner function (fn) has dtype float64. This can happen if the inner function of scan results in an upcast or downcast.

— Reply to this email directly or view it on GitHub https://github.com/lisa-groundhog/GroundHog/issues/28#issuecomment-84964502 .

[tachim]

I ran into this too. Make sure your floatX flag is consistent between runs.

[antxa]

Right. Setting floatX=float32 in Theano everything works OK. Thanks!