search

rizkiarm / LipNet

Keras implementation of 'LipNet: End-to-End Sentence-level Lipreading'
MIT License
635 stars 226 forks source link

a question about the net... #8

Closed kobenaxie closed 7 years ago

kobenaxie commented 7 years ago

code in model2.py, line 52: self.resh1 = TimeDistributed(Flatten())(self.maxp3)

Could you please tell me the input shape and out shape ?the shape of self.maxp3 and self.resh1 ?? Thank u~

rizkiarm commented 7 years ago

Hi @kobenaxie, You can run the model and see the input and output shape of each layer in the model summary.

kobenaxie commented 7 years ago

Hi @rizkiarm , Thankt you for your reply, i try to train a model, but does the information 'Video /LipNet/training/unseen_speakers/datasets/train-s/s4/pwbx6s.mpg has incorrect shape (74, 100, 50, 3), must be (75, 100, 50, 3) will inffect the training process ?

rizkiarm commented 7 years ago

Hi, it means that the video has a wrong dimension. By default, the training would still be performed normally with the omission of that video.

kobenaxie commented 7 years ago

OK, I will try latter,Thank you so much~

kobenaxie commented 7 years ago

Hi @rizkiarm , i run the code to train and set curriculum=None in generator, but get error like this Epoch 1/1 Process Process-2: Traceback (most recent call last): File "/home/jlxie/anaconda2/envs/tf1.1-py2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/home/jlxie/anaconda2/envs/tf1.1-py2/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, *self._kwargs) File "/home/jlxie/anaconda2/envs/tf1.1-py2/lib/python2.7/site-packages/keras/engine/training.py", line 606, in data_generator_task Traceback (most recent call last): File "/home/jlxie/github/LipNet/training/unseen_speakers/train.py", line 77, in train(run_name, 0, 1, 3, 100, 50, 75, 32, 50) File "/home/jlxie/github/LipNet/training/unseen_speakers/train.py", line 72, in train pickle_safe=True) File "/home/jlxie/anaconda2/envs/tf1.1-py2/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 88, in wrapper return func(args, **kwargs) File "/home/jlxie/anaconda2/envs/tf1.1-py2/lib/python2.7/site-packages/keras/engine/training.py", line 1851, in fit_generator str(generator_output)) ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None

I do not know how to deal it with .

rizkiarm commented 7 years ago

Hi @kobenaxie, it usually happened when there is some error in loading the data. You may check whether you have correctly configured the dataset folder or whether any code produced some error which results in generator failing.

rizkiarm commented 7 years ago

I'll close this issue for now. Please reopen it if you have any updates or more details.

riya-maan commented 5 years ago

@rizkiarm I have similar issue on training random_speakers. I think that I have configured the dataset folder properly. This is the error I'm getting:

$ ./train random_split Using GPUs Using TensorFlow backend.

Loading dataset list from cache... Found 796 videos for training. Found 199 videos for validation.

WARNING:tensorflow:From /Users/riyamaan/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1062: calling reduce_prod (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /Users/riyamaan/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1044: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /Users/riyamaan/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1008: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead

Layer (type) Output Shape Param #

the_input (InputLayer) (None, 75, 100, 50, 3) 0

zero1 (ZeroPadding3D) (None, 77, 104, 54, 3) 0

conv1 (Conv3D) (None, 75, 50, 25, 32) 7232

batc1 (BatchNormalization) (None, 75, 50, 25, 32) 128

actv1 (Activation) (None, 75, 50, 25, 32) 0

spatial_dropout3d_1 (Spatial (None, 75, 50, 25, 32) 0

max1 (MaxPooling3D) (None, 75, 25, 12, 32) 0

zero2 (ZeroPadding3D) (None, 77, 29, 16, 32) 0

conv2 (Conv3D) (None, 75, 25, 12, 64) 153664

batc2 (BatchNormalization) (None, 75, 25, 12, 64) 256

actv2 (Activation) (None, 75, 25, 12, 64) 0

spatial_dropout3d_2 (Spatial (None, 75, 25, 12, 64) 0

max2 (MaxPooling3D) (None, 75, 12, 6, 64) 0

zero3 (ZeroPadding3D) (None, 77, 14, 8, 64) 0

conv3 (Conv3D) (None, 75, 12, 6, 96) 165984

batc3 (BatchNormalization) (None, 75, 12, 6, 96) 384

actv3 (Activation) (None, 75, 12, 6, 96) 0

spatial_dropout3d_3 (Spatial (None, 75, 12, 6, 96) 0

max3 (MaxPooling3D) (None, 75, 6, 3, 96) 0

time_distributed_1 (TimeDist (None, 75, 1728) 0

bidirectional_1 (Bidirection (None, 75, 512) 3048960

bidirectional_2 (Bidirection (None, 75, 512) 1181184

dense1 (Dense) (None, 75, 28) 14364

softmax (Activation) (None, 75, 28) 0

Total params: 4,572,156.0 Trainable params: 4,571,772.0 Non-trainable params: 384.0

WARNING:tensorflow:From /Users/riyamaan/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1123: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead 2019-01-26 20:27:09.521736: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA Epoch 1/20 Process Process-1: Traceback (most recent call last): File "/Users/riyamaan/Desktop/LipNet/training/random_split/train.py", line 73, in train(run_name, 0, 20, 3, 100, 50, 75, 32, 50) File "/Users/riyamaan/Desktop/LipNet/training/random_split/train.py", line 69, in train pickle_safe=True) File "/Users/riyamaan/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 88, in wrapper return func(*args, **kwargs) File "/Users/riyamaan/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 1851, in fit_generator str(generator_output)) ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None

Can you please help me resolve this issue?