tdeboissiere / DeepLearningImplementations

Implementation of recent Deep Learning papers
MIT License
1.81k stars 650 forks source link

Error with WGAN, tensorflow backend and batch size > 32 #22

Closed engharat closed 7 years ago

engharat commented 7 years ago

Hi, I' m testing your WGAN on MNIST. with theano everything is ok, then today I've tried tensorflow backend with subpixel generator. Using 32 or lower batch size is working, but if I try 64/128 batch size I get this error: InvalidArgumentError (see above for traceback): Conv2DSlowBackpropInput: input and out_backprop must have the same batch size [[Node: conv2d_transpose_4 = Conv2DBackpropInput[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](conv2d_transpose_4/output_shape, transpose_4, Relu_12)]]

Here is the current namespace and parameters: Namespace(backend='tensorflow', batch_size=64, bn_mode=2, clamp_lower=-0.01, clamp_upper=0.01, disc_iterations=20, discriminator='discriminator', dset='mnist', generator='subpixel', img_dim=64, lr_D=5e-05, lr_G=5e-05, n_batch_per_epoch=200, nb_epoch=400, noise_dim=100, noise_scale=0.5, opt_D='RMSprop', opt_G='RMSprop', use_mbd=False)

Experiment parameters: opt_G RMSprop opt_D RMSprop noise_dim 100 noise_scale 0.5 generator subpixel disc_iterations 20 nb_epoch 400 bn_mode 2 batch_size 64 n_batch_per_epoch 200 image_dim_ordering tf lr_G 5e-05 img_dim 64 lr_D 5e-05 dset mnist discriminator discriminator use_mbd False clamp_lower -0.01 clamp_upper 0.01

tdeboissiere commented 7 years ago

Probably due to a hard coded batch size somewhere. Check that every call in the pipeline uses the same batch size. The other option is a keras implementation error (less likely). I'll try and fix that asap.

tdeboissiere commented 7 years ago

Fixed it. In data_utils.py, I was using generator_model.predict() without the batch size argument (which defaults to 32) hence the error message.