talmo / leap

LEAP is now deprecated -- check out its successor SLEAP!
https://sleap.ai
Apache License 2.0
206 stars 48 forks source link

error with fast training #8

Closed epetter closed 5 years ago

epetter commented 5 years ago

Hello,

I'm excited to use your programs and had no problems with the demo dataset. I am trying to label 2 points on a mouse (head and base of tail) moving in an open field as an exercise. I am running into a problem with the fast training part of the label_joints program. It will generate the following error:

"Error when checking target: expected conv2d_12 to have shape (112, 112, 2) but got array with shape (110, 110, 2)"

I'm pretty new to python so it is hard for me to figure out where the error may originate. it seems like it is within the keras scripts

if it helps, i am including the output from matlab.

Please let me know if I can provide any more information.

Thank you, Konstantin Bakhurin

Found 16/80 labeled frames. Loaded 16 images [0.08s] Generated confidence maps [0.15s] 1.477 MB Saved: C:\Users\Yinlab\AppData\Local\Temp\tpd285a349_2f4f_445a_9550_3716b419b82b.h5 [0.16s] tpd285a349_2f4f_445a_9550_3716b419b82b.h5: 2.96 MB Finished generating training set. [0.69s] session timestamp message


   1       01-Oct-2018 17:12:13    "Fast training (n = 16)"

python "C:\Users\Yinlab\Documents\GitHub\leap\leap\training.py" "C:\Users\Yinlab\AppData\Local\Temp\tpd285a349_2f4f_445a_9550_3716b419b82b.h5" --base-output-path="C:\Data\Open Field Data\pythonTest\stimulation\models" --run-name="181001_171205-n=16" --net-name="leap_cnn" --filters=32 --rotate-angle=0 --val-size=0.10000 --epochs=15 --batch-size=50 --batches-per-epoch=50 --val-batches-per-epoch=10 --reduce-lr-factor=0.1000000000 --reduce-lr-patience=2 --reduce-lr-cooldown=0 --reduce-lr-min-delta=0.0000100000 --reduce-lr-min-lr=0.0000000001 --upsampling-layers --amsgrad 2018-10-01 17:12:16.424964: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2 C:\Users\Yinlab\Anaconda3\lib\site-packages\h5py__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Using TensorFlow backend. C:\Users\Yinlab\Anaconda3\lib\site-packages\keras\callbacks.py:999: UserWarning: epsilon argument is deprecated and will be removed, use min_delta instead. warnings.warn('epsilon argument is deprecated and ' data_path: C:\Users\Yinlab\AppData\Local\Temp\tpd285a349_2f4f_445a_9550_3716b419b82b.h5 Loaded 16 samples [0.0s] Permuted and normalized data. [0.0s] box.shape: (14, 110, 110, 1) val_box.shape: (2, 110, 110, 1) img_size: (110, 110, 1) num_output_channels: 2 data_name: tpd285a349_2f4f_445a_9550_3716b419b82b run_name: 181001_171205-n=16


Layer (type) Output Shape Param #
================================================================= input_1 (InputLayer) (None, 110, 110, 1) 0


conv2d_1 (Conv2D) (None, 110, 110, 32) 320


conv2d_2 (Conv2D) (None, 110, 110, 32) 9248


conv2d_3 (Conv2D) (None, 110, 110, 32) 9248


max_pooling2d_1 (MaxPooling2 (None, 55, 55, 32) 0


conv2d_4 (Conv2D) (None, 55, 55, 64) 18496


conv2d_5 (Conv2D) (None, 55, 55, 64) 36928


conv2d_6 (Conv2D) (None, 55, 55, 64) 36928


max_pooling2d_2 (MaxPooling2 (None, 28, 28, 64) 0


conv2d_7 (Conv2D) (None, 28, 28, 128) 73856


conv2d_8 (Conv2D) (None, 28, 28, 128) 147584


conv2d_9 (Conv2D) (None, 28, 28, 128) 147584


up_sampling2d_1 (UpSampling2 (None, 56, 56, 128) 0


conv2d_10 (Conv2D) (None, 56, 56, 64) 73792


conv2d_11 (Conv2D) (None, 56, 56, 64) 36928


up_sampling2d_2 (UpSampling2 (None, 112, 112, 64) 0


conv2d_12 (Conv2D) (None, 112, 112, 2) 1154
================================================================= Total params: 592,066 Trainable params: 592,066 Non-trainable params: 0


Created folder: C:\Data\Open Field Data\pythonTest\stimulation\models\181001_171205-n=16 Epoch 1/15 Traceback (most recent call last): File "C:\Users\Yinlab\Documents\GitHub\leap\leap\training.py", line 272, in clize.run(train) File "C:\Users\Yinlab\Anaconda3\lib\site-packages\sigtools\modifiers.py", line 158, in call return self.func(*args, kwargs) File "C:\Users\Yinlab\Anaconda3\lib\site-packages\clize\runner.py", line 360, in run ret = cli(args) File "C:\Users\Yinlab\Anaconda3\lib\site-packages\clize\runner.py", line 220, in call return func(posargs, kwargs) File "C:\Users\Yinlab\Documents\GitHub\leap\leap\training.py", line 251, in train viz_grid_callback File "C:\Users\Yinlab\Anaconda3\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "C:\Users\Yinlab\Anaconda3\lib\site-packages\keras\engine\training.py", line 1426, in fit_generator initial_epoch=initial_epoch) File "C:\Users\Yinlab\Anaconda3\lib\site-packages\keras\engine\training_generator.py", line 191, in fit_generator class_weight=class_weight) File "C:\Users\Yinlab\Anaconda3\lib\site-packages\keras\engine\training.py", line 1214, in train_on_batch class_weight=class_weight) File "C:\Users\Yinlab\Anaconda3\lib\site-packages\keras\engine\training.py", line 792, in _standardize_user_data exception_prefix='target') File "C:\Users\Yinlab\Anaconda3\lib\site-packages\keras\engine\training_utils.py", line 136, in standardize_input_data str(data_shape)) ValueError: Error when checking target: expected conv2d_12 to have shape (112, 112, 2) but got array with shape (110, 110, 2)

talmo commented 5 years ago

Hi Kostantin,

The issue here seems to be the size of your images. The input size now appears to be 110x110, but you'll need to resize, crop or pad to a multiple of 4.

108 / 4 = 27 -> Works 110 / 4 = 27.5 -> Doesn't work 112 / 4 = 28 -> Works

So you could either crop a pixel off of each side or pad each side with a 0 and it should solve the problem.

Let me know if you're still having issues after doing that though!

Cheers,

Talmo

kbakhurin commented 5 years ago

Hi Talmo,

Thanks for your help! That solved it.

If I have further questions about using the software, should I start a new issue or email you directly?

Thanks, Konstantin