search

Justin-Tan / generative-compression

TensorFlow Implementation of Generative Adversarial Networks for Extreme Learned Image Compression
MIT License
511 stars 108 forks source link

A problem when training with other dataset #28

Closed Lishiyuan0813 closed 5 years ago

Lishiyuan0813 commented 5 years ago

The size of my dataset is 640*480 and the width and length are all multiple of 16. But there is an error as below.

Training on dataset cityscapes Building computational graph ... Training on cityscapes Training on cityscapes <------------ Building global image generator architecture ------------> Sampling noise... Real image shape: [None, None, None, 3] Reconstruction shape: [None, 512, 1024, 3] <------------ Building multiscale discriminator architecture ------------> Building discriminator D(x) Shape of x: [None, None, None, 3] Shape of x downsampled by factor 2: [None, None, None, 3] Shape of x downsampled by factor 4: [None, None, None, 3] <------------ Building multiscale discriminator architecture ------------> Building discriminator D(G(z)) Shape of x: [None, 512, 1024, 3] Shape of x downsampled by factor 2: [None, 256, 512, 3] Shape of x downsampled by factor 4: [None, 128, 256, 3] 2019-01-25 20:51:19.330069: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2019-01-25 20:51:19.468798: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: name: GeForce GTX TITAN X major: 5 minor: 2 memoryClockRate(GHz): 1.076 pciBusID: 0000:02:00.0 totalMemory: 11.92GiB freeMemory: 11.64GiB 2019-01-25 20:51:19.468827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0 2019-01-25 20:51:23.913861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-01-25 20:51:23.913922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0 2019-01-25 20:51:23.913936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N 2019-01-25 20:51:23.932503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11267 MB memory) -> physical GPU (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:02:00.0, compute capability: 5.2) Traceback (most recent call last): File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call return fn(*args) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape[0] = [1,8,30,40] vs. shape[1] = [1,8,32,64] [[Node: generator/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](generator/concat-0-TransposeNHWCToNCHW-LayoutOptimizer, generator/noise_generator/conv_out/conv2d/BiasAdd, generator/concat-2-LayoutOptimizer)]] [[Node: generator/decoder/conv2d_transpose_1/Shape/_65 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1756_generator/decoder/conv2d_transpose_1/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train.py", line 119, in main() File "train.py", line 116, in main train(config_train, args) File "train.py", line 70, in train start_time, epoch, args.name, G_loss_best, D_loss_best) File "/home/gwh/Desktop/generative-compression-master/utils.py", line 78, in run_diagnostics G_loss, D_loss, summary = sess.run([model.G_loss, model.D_loss, model.merge_op], feed_dict=feed_dict_test) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run run_metadata_ptr) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run run_metadata) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: ConcatOp : Dimensions of inputs should match: shape[0] = [1,8,30,40] vs. shape[1] = [1,8,32,64] [[Node: generator/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](generator/concat-0-TransposeNHWCToNCHW-LayoutOptimizer, generator/noise_generator/conv_out/conv2d/BiasAdd, generator/concat-2-LayoutOptimizer)]] [[Node: generator/decoder/conv2d_transpose_1/Shape/_65 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1756_generator/decoder/conv2d_transpose_1/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'generator/concat', defined at: File "train.py", line 119, in main() File "train.py", line 116, in main train(config_train, args) File "train.py", line 34, in train gan = Model(config, paths, name=args.name, dataset=args.dataset) File "/home/gwh/Desktop/generative-compression-master/model.py", line 77, in init self.z = tf.concat([self.w_hat, Gv], axis=-1) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1189, in concat return gen_array_ops.concat_v2(values=values, axis=axis, name=name) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 953, in concat_v2 "ConcatV2", values=values, axis=axis, name=name) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op op_def=op_def) File "/home/gwh/anaconda2/envs/lishiyuan/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): ConcatOp : Dimensions of inputs should match: shape[0] = [1,8,30,40] vs. shape[1] = [1,8,32,64] [[Node: generator/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](generator/concat-0-TransposeNHWCToNCHW-LayoutOptimizer, generator/noise_generator/conv_out/conv2d/BiasAdd, generator/concat-2-LayoutOptimizer)]] [[Node: generator/decoder/conv2d_transpose_1/Shape/_65 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1756_generator/decoder/conv2d_transpose_1/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

2019-01-25 20:51:33.325745: E tensorflow/core/kernels/data/cache_dataset_ops.cc:314] The calling iterator did not fully read the dataset we were attempting to cache. In order to avoid unexpected truncation of the sequence, the current [partially cached] sequence will be dropped. This can occur if you have a sequence similar to dataset.cache().take(k).repeat(). Instead, swap the order (i.e. dataset.take(k).cache().repeat())

gullibleretorted commented 3 years ago

Hi Lishiyuan0813,

During Inference I am facing the same error your have here at the end:

2019-01-25 20:51:33.325745: E tensorflow/core/kernels/data/cache_dataset_ops.cc:314] The calling iterator did not fully read the dataset we were attempting to cache. In order to avoid unexpected truncation of the sequence, the current [partially cached] sequence will be dropped. This can occur if you have a sequence similar to dataset.cache().take(k).repeat(). Instead, swap the order (i.e. dataset.take(k).cache().repeat())

Did you, by any chance, end up with a solution for that problem?

Thanks a lot in advance!