Seya needs to be updated for Keras 0.2.0

[daquang]

For example, the Bidirectional layer needs to define output_dim now. Also, layers such as Dense no longer have input_dim as a required argument, and the first argument now corresponds to the output_dim

[EderSantana]

I believe I covered almost everything with this commit https://github.com/EderSantana/seya/commit/71297e4c330b4075144c195ba881d8a82c5cf6c6

Let me know if something was left out

[amitbeka]

I have tried to use the Bidirectional example with IMDB, and it fails with some theano problem. I've checked it both with the latest keras version and the version tagged as 0.2.0, same result:

Loading data...
20000 train sequences
5000 test sequences
Pad sequences (samples x time)
X_train shape: (20000, 100)
X_test shape: (5000, 100)
Build model...
Train...
Train on 20000 samples, validate on 5000 samples
Epoch 1/4
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/home/beka/dev/seya/examples/imdb_brnn.py in <module>()
     66 
     67 print("Train...")
---> 68 model.fit(X_train, y_train, batch_size=batch_size, nb_epoch=4, validation_data=(X_test, y_test), show_accuracy=True)
     69 score, acc = model.evaluate(X_test, y_test, batch_size=batch_size, show_accuracy=True)
     70 print('Test score:', score)

/home/beka/dev/keras/keras/models.py in fit(self, X, y, batch_size, nb_epoch, verbose, callbacks, validation_split, validation_data, shuffle, show_accuracy, class_weight, sample_weight)
    492                          verbose=verbose, callbacks=callbacks,
    493                          val_f=val_f, val_ins=val_ins,
--> 494                          shuffle=shuffle, metrics=metrics)
    495 
    496     def predict(self, X, batch_size=128, verbose=0):

/home/beka/dev/keras/keras/models.py in _fit(self, f, ins, out_labels, batch_size, nb_epoch, verbose, callbacks, val_f, val_ins, shuffle, metrics)
    213                 batch_logs['size'] = len(batch_ids)
    214                 callbacks.on_batch_begin(batch_index, batch_logs)
--> 215                 outs = f(*ins_batch)
    216                 if type(outs) != list:
    217                     outs = [outs]

/home/beka/venv/eva/lib/python2.7/site-packages/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    604                         self.fn.nodes[self.fn.position_of_error],
    605                         self.fn.thunks[self.fn.position_of_error],
--> 606                         storage_map=self.fn.storage_map)
    607                 else:
    608                     # For the c linker We don't have access from

/home/beka/venv/eva/lib/python2.7/site-packages/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    593         t0_fn = time.time()
    594         try:
--> 595             outputs = self.fn()
    596         except Exception:
    597             if hasattr(self.fn, 'position_of_error'):

ValueError: Input dimension mis-match. (input[0].shape[1] = 128, input[1].shape[1] = 64)
Apply node that caused the error: Elemwise{mul,no_inplace}(Join.0, Elemwise{Composite{Cast{float64}(LT(i0, i1))}}.0)
Inputs types: [TensorType(float64, matrix), TensorType(float64, matrix)]
Inputs shapes: [(32, 128), (32, 64)]
Inputs strides: [(1024, 8), (512, 8)]
Inputs values: ['not shown', 'not shown']

Backtrace when the node is created:
  File "/home/beka/dev/keras/keras/layers/core.py", line 416, in get_output
    X *= self.srng.binomial(X.shape, p=retain_prob, dtype=theano.config.floatX)

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

[EderSantana]

@amitbeka this seem to be an error I had fixed, the output shape was half of the correct value... also, that example is working for me. Can you please check which commit of Seya are you in? Use this code inside Seya folder
git log --pretty=format:'%h' -n 1

[amitbeka]

I think I'm using the latest commit: 0b8fd60 in keras i'm on commit 47071f9 (also latest). tried to update theano to the latest version but that didn't help either

[EderSantana]

I just ran the experiment in my computer with success. Do you mind creating a new virtual env and installing keras+seya to double check? I honestly think it should be working.

This error was caused because the output_dim being passed by the shape inference was half the actual size. But I fixed that...

[amitbeka]

just did it (new virtualenv with pip install git+git://github.com/fchollet/keras and manually git clone seya), and still the same result.

I think it has something to do with python versions... The above error happens on python2.7.10.

I tried also a virtualenv with python 3.5, and got a different error:

PYTHONPATH=. python examples/imdb_brnn.py      
Loading data...
20000 train sequences
5000 test sequences
Pad sequences (samples x time)
X_train shape: (20000, 100)
X_test shape: (5000, 100)
Build model...
Traceback (most recent call last):
  File "examples/imdb_brnn.py", line 65, in <module>
    model.compile(loss='binary_crossentropy', optimizer='adam', class_mode="binary")
  File "/home/beka/venv/seya/lib/python3.5/site-packages/keras/models.py", line 358, in compile
    self.y_train = self.get_output(train=True)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/keras/layers/containers.py", line 74, in get_output
    return self.layers[-1].get_output(train)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/keras/layers/core.py", line 439, in get_output
    X = self.get_input(train)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/keras/layers/core.py", line 102, in get_input
    return self.previous.get_output(train=train)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/keras/layers/core.py", line 613, in get_output
    X = self.get_input(train)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/keras/layers/core.py", line 102, in get_input
    return self.previous.get_output(train=train)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/keras/layers/core.py", line 412, in get_output
    X = self.get_input(train)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/keras/layers/core.py", line 102, in get_input
    return self.previous.get_output(train=train)
  File "/tmp/seya/seya/layers/recurrent.py", line 71, in get_output
    Xf = self.forward.get_output(train)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/keras/layers/recurrent.py", line 446, in get_output
    T.unbroadcast(alloc_zeros_matrix(X.shape[1], self.output_dim), 1),
  File "/home/beka/venv/seya/lib/python3.5/site-packages/keras/utils/theano_utils.py", line 28, in alloc_zeros_matrix
    return T.alloc(np.cast[theano.config.floatX](0.), *dims)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/theano/tensor/basic.py", line 2546, in __call__
    ret = super(Alloc, self).__call__(val, *shapes, **kwargs)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/theano/gof/op.py", line 507, in __call__
    node = self.make_node(*inputs, **kwargs)
  File "/home/beka/venv/seya/lib/python3.5/site-packages/theano/tensor/basic.py", line 2424, in make_node
    (i, s_as_str))
TypeError: Shape arguments to Alloc must be integers, but argument 1 is not for apply node: TensorConstant{64.0}

And from what i see, there is still a small error with the output_dim variable, on recurrent.py, line 26: self.output_dim = self.forward.output_dim which should be self.forward.output_dim*2 if i'm not mistaken...

Thanks for the quick response (and open-sourcing your work :))!

[EderSantana]

@amitbeka tkx for taking your time to report your results.

I see that ouput_dim is wrong, but note that the output_shape was actually calculated as https://github.com/EderSantana/seya/blob/master/seya/layers/recurrent.py#L60-L68

Also your python3 error is probably because of the true division. That can be fixed easily.

Anyway, I just made a commit that hopefully will get you covered https://github.com/EderSantana/seya/commit/bb88053c1f09ef1a0414ade769054f0cd33a087c

[amitbeka]

Unfortunately, it only makes python3 error the same as python2, so I still have a dimension mis-match error when running fit().

Does it happen in your setup if you fresh-install a new environment? I wonder what's happening here

[EderSantana]

@amitbeka I was able to reproduce the error with a clean conda env. I'll check carefully what is happening. If you find out anything, please let me know.

[EderSantana]

@amitbeka the problem was with the old Theano that conda provides. Try update Theano as:

$ conda (or pip) uninstall theano
$ pip install git+https://github.com/Theano/Theano.git@15c90dd3#egg=Theano==0.8.git

I'm gonna update Seya requirements.

[amitbeka]

I just found out that with theano_mode=FAST_COMPILE it doesn't happen, so probably theano. I will validate in the morning. thanks for all the help!

[EderSantana]

I think it is something with T.concatenate.

[amitbeka]

Just validated it, and it works great, also in more complex settings (masking). Thanks for the help again

[EderSantana]

tkx for the feedback @amitbeka