Closed lxy-94 closed 6 years ago
@lxyhahaha did you solve the problem?
@anas-899 I solved it by following code:
@WenbinYang123
Thanks a lot for your fast response.
I tried your solution and the previous error disappear but I got another error:
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[512] [[Node: bn3b_branch2c/moments/mean = Mean[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=true, _device="/job:localhost/replica:0/task:0/gpu:0"](res3b_branch2c/BiasAdd, bn3b_branch2c/moments/mean/reduction_indices)]] [[Node: loss/add_53/_2497 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_28257_loss/add_53", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[512] [[Node: bn3b_branch2c/moments/mean = Mean[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=true, _device="/job:localhost/replica:0/task:0/gpu:0"](res3b_branch2c/BiasAdd, bn3b_branch2c/moments/mean/reduction_indices)]] [[Node: loss/add_53/_2497 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_28257_loss/add_53", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
do you know how can I resolve it?
@WenbinYang123
it turned out that happened because of big batch_size = 16 I changed it to batch_size = 4 and it worked without any problem.
Thanks
@WenbinYang123 I am getting new error after the training started:
raise ValueError('If printing histograms, validation_data must be ' ValueError: If printing histograms, validation_data must be provided, and cannot be a generator.
this is happening at the end of first Epoch 1/200 2781/2782 [============================>.] - ETA: 0s - loss: 0.9863 - sparse_accuracy_ignoring_last_label: 0.7550Traceback
are you getting same error?
@anas-899 you can change histogram_freq=10 to histogram_freq=0.
@lxyhahaha you seem to be using Python 3 where /
does floating point division by default; you might want to change that to //
which performs integral division.
@jermenkoo Thank you, I have just solved it by following your suggest.
Merged a PR that should fix this, let me know if there are still problems, thanks!
Hello, thank you for your code. When I try to run the train.py to do a demo, but I get a "TypeError:
pad_width
must be of integral type.", and it points to the "Keras-FCN\utils\SegDataGenerator.py", line 238, in next x = np.lib.pad(x, ((pad_h / 2, pad_h - pad_h / 2), (pad_w / 2, pad_w - pad_w / 2), (0, 0)), 'constant', constant_values=0.)". I don't know how to fix this problem, can you give me some suggestions?