Sample example code in your blog not working and giving TypeError: Cannot convert Type TensorType?

[ghost]

I have came across your tutorial (http://benjaminbolte.com/blog/2016/keras-language-modeling.html) in the web and its pretty simple and easy to understand. As a begineer I tried out your first sample code in my keras (version 1.2.0) with theano backend

import itertools
import numpy as np

sentences = '''
sam is red
hannah not red
hannah is green
bob is green
bob not red
sam not green
sarah is red
sarah not green'''.strip().split('\n')
is_green = np.asarray([[0, 1, 1, 1, 1, 0, 0, 0]], dtype='int32').T

lemma = lambda x: x.strip().lower().split(' ')
sentences_lemmatized = [lemma(sentence) for sentence in sentences]
words = set(itertools.chain(*sentences_lemmatized))
# set(['boy', 'fed', 'ate', 'cat', 'kicked', 'hat'])

# dictionaries for converting words to integers and vice versa
word2idx = dict((v, i) for i, v in enumerate(words))
idx2word = list(words)

# convert the sentences a numpy array
to_idx = lambda x: [word2idx[word] for word in x]
sentences_idx = [to_idx(sentence) for sentence in sentences_lemmatized]
sentences_array = np.asarray(sentences_idx, dtype='int32')

# parameters for the model
sentence_maxlen = 3
n_words = len(words)
n_embed_dims = 3

# put together a model to predict 
from keras.layers import Input, Embedding, merge, Flatten, SimpleRNN
from keras.models import Model

input_sentence = Input(shape=(sentence_maxlen,), dtype='int32')
input_embedding = Embedding(n_words, n_embed_dims)(input_sentence)
color_prediction = SimpleRNN(1)(input_embedding)

predict_green = Model(input=[input_sentence], output=[color_prediction])
predict_green.compile(optimizer='sgd', loss='binary_crossentropy')

# fit the model to predict what color each person is
predict_green.fit([sentences_array], [is_green], nb_epoch=5000, verbose=1)
embeddings = predict_green.layers[1].W.get_value()

# print out the embedding vector associated with each word
for i in range(n_words):
    print('{}: {}'.format(idx2word[i], embeddings[i]))

But at this line

# fit the model to predict what color each person is
predict_green.fit([sentences_array], [is_green], nb_epoch=5000, verbose=1)
embeddings = predict_green.layers[1].W.get_value()

I am getting this error

TypeError: Cannot convert Type TensorType(float32, 3D) (of Variable Subtensor{:int64:}.0) into Type TensorType(float32, (False, False, True)). You can try to manually convert Subtensor{:int64:}.0 into a TensorType(float32, (False, False, True)).

[codekansas]

Try using a different Python version (I remember having this error if I used Python 2.7 or something).

It's a bug with Theano I believe.

[ghost]

My python is 2.7 and theano is 0.8.2. Is there any other work around other than using different version of python?

[codekansas]

Yes, you can make is_green an (8, 2) matrix by doubling all the outputs. It only seems to happen with python 2.7 / theano when the last output dimension is 1.

[ghost]

Hi @codekansas From your suggestion I was able to run the script in my system successfully. These are the changes I made; I changed the is_green matrix to (8,2) by repeating the same output once more and also changed the output dimension of SimpleRNN function

is_green = np.asarray([[0, 1, 1, 1, 1, 0, 0, 0],
                       [0, 1, 1, 1, 1, 0, 0, 0]], dtype='int32').T

color_prediction = SimpleRNN(2)(input_embedding)

I really dont know whether my logic is correct or not but its outputing result

sarah: [ 0.03456502 -0.00972695 -0.04076455]
sam: [ 0.04495115 -0.02211441  0.02328646]
hannah: [-0.03459289 -0.05863173 -0.08748473]
is: [-0.00041237 -0.01037021 -0.02777916]
green: [-0.01604047  0.01044582 -0.00402654]
not: [ 0.02608304 -0.0525613   0.01888898]
bob: [ 0.00324048  0.05128151 -0.08478668]
red: [-0.00443861  0.03307148 -0.0740037 ]

Full code

import itertools
import numpy as np

sentences = '''
sam is red
hannah not red
hannah is green
bob is green
bob not red
sam not green
sarah is red
sarah not green'''.strip().split('\n')
is_green = np.asarray([[0, 1, 1, 1, 1, 0, 0, 0],
                       [0, 1, 1, 1, 1, 0, 0, 0]], dtype='int32').T

lemma = lambda x: x.strip().lower().split(' ')
sentences_lemmatized = [lemma(sentence) for sentence in sentences]
words = set(itertools.chain(*sentences_lemmatized))
# set(['boy', 'fed', 'ate', 'cat', 'kicked', 'hat'])

# dictionaries for converting words to integers and vice versa
word2idx = dict((v, i) for i, v in enumerate(words))
idx2word = list(words)

# convert the sentences a numpy array
to_idx = lambda x: [word2idx[word] for word in x]
sentences_idx = [to_idx(sentence) for sentence in sentences_lemmatized]
sentences_array = np.asarray(sentences_idx, dtype='int32')

# parameters for the model
sentence_maxlen = 3
n_words = len(words)
n_embed_dims = 3

# put together a model to predict 
from keras.layers import Input, Embedding, merge, Flatten, SimpleRNN
from keras.models import Model

input_sentence = Input(shape=(sentence_maxlen,), dtype='int32')
input_embedding = Embedding(n_words, n_embed_dims)(input_sentence)
color_prediction = SimpleRNN(2)(input_embedding)

predict_green = Model(input=[input_sentence], output=[color_prediction])
predict_green.compile(optimizer='sgd', loss='binary_crossentropy')

# fit the model to predict what color each person is
predict_green.fit([sentences_array], [is_green], nb_epoch=5, verbose=1)
embeddings = predict_green.layers[1].W.get_value()

# print out the embedding vector associated with each word
for i in range(n_words):
    print('{}: {}'.format(idx2word[i], embeddings[i]))

[codekansas]

That seems right. You might have to mess with the training parameters or turn off RNN training so that the embeddings look like the graph I included