dtype int16 instead of dtype int8 in outputs_info

[moty66]

ValueError: When compiling the inner function of scan the following error has been encountered: The initial state (outputs_info in scan nomenclature) of variable IncSubtensor{Set;:int64:}.0 (argument number 3) has dtype int16, while the result of the inner function (fn) has dtype int8. This can happen if the inner function of scan results in an upcast or downcast.

OSX 10.11.6 CUDA Driver Version: 8.0.46 python 2.7

[moty66]

Fulltrace:

Using gpu device 0: GeForce GTX 675MX (CNMeM is enabled with initial size: 24.0% of memory, cuDNN 5105)
Loaded TEST_MIDI1
Traceback (most recent call last):
  File "main.py", line 43, in <module>
    m = model.Model([300,300],[100,50], dropout=0.5)
  File "/Users/*/neural/biaxial-rnn-music-composition/model.py", line 101, in __init__
    self.setup_predict()
  File "/Users/*/neural/biaxial-rnn-music-composition/model.py", line 315, in setup_predict
    n_steps=self.steps_to_simulate )
  File "/usr/local/lib/python2.7/site-packages/theano/scan_module/scan.py", line 761, in scan
    condition, outputs, updates = scan_utils.get_updates_and_outputs(fn(*args))
  File "/Users/*/neural/biaxial-rnn-music-composition/model.py", line 295, in step_time
    notes_result, updates = theano.scan(fn=self._predict_step_note, sequences=[time_final], outputs_info=note_outputs_info)
  File "/usr/local/lib/python2.7/site-packages/theano/scan_module/scan.py", line 1058, in scan
    scan_outs = local_op(*scan_inputs)
  File "/usr/local/lib/python2.7/site-packages/theano/gof/op.py", line 604, in __call__
    node = self.make_node(*inputs, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/theano/scan_module/scan_op.py", line 539, in make_node
    inner_sitsot_out.type.dtype))
ValueError: When compiling the inner function of scan the following error has been encountered: The initial state (`outputs_info` in scan nomenclature) of variable IncSubtensor{Set;:int64:}.0 (argument number 3) has dtype int16, while the result of the inner function (`fn`) has dtype int8. This can happen if the inner function of scan results in an upcast or downcast.

[ZhyiXu]

I ran 'python train.py' and the same problem happened, the raise error is the same. Is there any solutions?

[moty66]

I forgot to mention that this happen only when the device is GPU, switching to CPU no error occorse.

Is it a theano issue?

[ZhyiXu]

I have no idea but if this model only train on CPU, the time maybe horrible long

[danieldjohnson]

My guess is that this was caused by something that changed in a recent release of Theano. I've added an explicit dtype for the offending allocation, which hopefully will fix it. Unfortunately I don't have access to a GPU system right now, so I can't easily test changes. Can you check if my most recent commit resolves the issue?

[danieldjohnson]

Closing for now, assuming fixed. Let me know if it still doesn't work after you pull the fix.

[moty66]

I confirm that the issue is fixed after adding the explicit dtype

Unfortunatly, this model is eating lot of memory and I can't finish the training because of "out of memory"

I also confirm that on CPU the training takes really too much time. On iMAC i7 with 16gb of ram for 3 midi files it take something like 5 days

Thank you for the fix

[tristatian]

ValueError: When compiling the inner function of scan the following error has been encountered: The initial state (outputs_info in scan nomenclature) of variable IncSubtensor{Set;:int64:}.0 (argument number 3) has dtype int8, while the result of the inner function (fn) has dtype bool. This can happen if the inner function of scan results in an upcast or downcast.

Python 3.7.1 OSX 10.14.4