Incompatible shapes

[vijtad]

I used import_meta_graph and restore and have used my own image.

I am getting this error

Incompatible shapes: [1,30,40,21] vs. [1,18,28,21] at

image_np, pred_np = sess.run([image_tensor, pred], feed_dict=feed_dict_to_use)

[warmspringwinds]

Hi,

Please, provide the full code that you run.

[ahundt]

Please correct the formatting of your post to be displayed as code: https://help.github.com/articles/creating-and-highlighting-code-blocks/

[vijtad]

I am using Python 3.5 running TF v012 on Windows 10. I don't see model_fcn8s_final.ckpt in your drop box and used meta data. Please review my code.

from __future__ import division

import os
import sys
import tensorflow as tf
import skimage.io as io
import numpy as np

sys.path.append("tf-image-segmentation/")
sys.path.append("/home/dpakhom1/workspace/my_models/slim/")

fcn_16s_checkpoint_path = \
 'C:/TensorFlow/checkpoints/vgg_16.ckpt'

os.environ["CUDA_VISIBLE_DEVICES"] = '1'

slim = tf.contrib.slim

from tf_image_segmentation.models.fcn_8s import FCN_8s
from tf_image_segmentation.utils.inference import adapt_network_for_any_size_input
from tf_image_segmentation.utils.pascal_voc import pascal_segmentation_lut

number_of_classes = 21

image_filename = 'C:/Tensorflow/sticker/me.jpg'

#image_filename = 'C:/Tensorflow/sticker/small_cat.jpg'

image_filename_placeholder = tf.placeholder(tf.string)

feed_dict_to_use = {image_filename_placeholder: image_filename}

image_tensor = tf.read_file(image_filename_placeholder)

image_tensor = tf.image.decode_jpeg(image_tensor, channels=3)

# Fake batch for image and annotation by adding
# leading empty axis.
image_batch_tensor = tf.expand_dims(image_tensor, axis=0)

# Be careful: after adaptation, network returns final labels
# and not logits
FCN_8s = adapt_network_for_any_size_input(FCN_8s, 32)

pred, fcn_16s_variables_mapping = FCN_8s(image_batch_tensor=image_batch_tensor,
                                          number_of_classes=number_of_classes,
                                          is_training=False)

# The op for initializing the variables.
initializer = tf.global_variables_initializer()

#saver = tf.train.Saver()
saver = tf.train.import_meta_graph('C:/TensorFlow/checkpoints/fcn_8s_checkpoint/model_fcn8s_final.ckpt.meta', clear_devices=True)

with tf.Session() as sess:

    sess.run(initializer)
    #saver.restore(sess,fcn_16s_checkpoint_path)
    saver.restore(sess, "C:/TensorFlow/checkpoints/fcn_8s_checkpoint/model_fcn8s_final.ckpt")
    image_np, pred_np = sess.run([image_tensor, pred], feed_dict=feed_dict_to_use)

    io.imshow(image_np)
    io.show()

    io.imshow(pred_np.squeeze())
    io.show()

[warmspringwinds]

@vijtad , I didn't test this code for python 3 yet. It makes it hard for me to understand what's the reason for your mistake.

If you solve it, feel free to post your workaround.

[naharsingh]

Hi @vijtad, @warmspringwinds, i am getting same issue with python 2.7. Would you please suggest solution.

[vijtad]

I highlighted my workaround to make it work in Python 3.5 as described in https://github.com/warmspringwinds/tf-image-segmentation/issues/11 But still segmentation is not perfect.

Waiting for your help on perfect segmentation.

[ahundt]

@vijtad Perfect segmentation remains an open research question. In other words, nobody has solved it yet. Here is a leaderboard where you can see what it means to have state of the art results: http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?challengeid=11&compid=6

[sebaschaal]

I have a similar problem.

I made no changes to your code, but I also get compatibility issues. I narrowed it down to the following:

fcn_8s.py line 131:
fused_last_layer_and_pool4_logits = pool4_logits + last_layer_upsampled_by_factor_2_logits

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,38,38,21] vs. [1,26,26,21]
[[Node: fcn_8s/add = Add[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](fcn_8s/pool4_fc/BiasAdd, fcn_8s/conv2d_transpose)]]

Some wiring is not done correctly in my view. I experimented with different input size:

• 616x616 --> Incompatible shapes: [1,38,38,21] vs. [1,26,26,21]

• 500x500 --> Incompatible shapes: [1,32,32,21] vs. [1,20,20,21]

• 598x800 --> Incompatible shapes: [1,38,50,21] vs. [1,26,38,21]

[naharsingh]

Hi Sebastian, I was getting same issue and resolved by using branch of vgg 'git clone -b fully_conv_vgg HTTPS://github.com/warm spring winds/models

Hope it will help.

On 23 Mar 2017 3:26 p.m., "Sebastian Schaal" notifications@github.com wrote:

I have a similar problem.

I made no changes to your code, but I also get compatibility issues. I narrowed it down to the following:

fcn_8s.py line 131: fused_last_layer_and_pool4_logits = pool4_logits + last_layer_upsampled_by_factor_2_logits

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,38,38,21] vs. [1,26,26,21] [[Node: fcn_8s/add = AddT=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

Some wiring is not done correctly in my view. I experimented with different input size:

616x616 --> Incompatible shapes: [1,38,38,21] vs. [1,26,26,21]

500x500 --> Incompatible shapes: [1,32,32,21] vs. [1,20,20,21]

598x800 --> Incompatible shapes: [1,38,50,21] vs. [1,26,38,21]

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/warmspringwinds/tf-image-segmentation/issues/4#issuecomment-288669635, or mute the thread https://github.com/notifications/unsubscribe-auth/AViHYGY1nAxVK6fnvxgROnFZIGuWx1vCks5rokHlgaJpZM4L7kzB .

[arasharchor]

I have changed nothing in running FCN_32s_train.py except changing image size and number of classes. image_train_size = [850, 850] number_of_classes = 2 I had created tfrecord successfully, but after running fcn_32s_train.py I get the following error: @vijtad @naharsingh @warmspringwinds @sebaschaal

Total memory: 11.90GiB
Free memory: 11.76GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:83:00.0)
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:247] PoolAllocator: After 44669 get requests, put_count=44143 evicted_count=1000 eviction_rate=0.0226536 and unsatisfied allocation rate=0.0364011
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:259] Raising pool_size_limit_ from 100 to 110
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed
Traceback (most recent call last):
  File "fcn_32s_train.py", line 145, in <module>
    train_step ])
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
    run_metadata_ptr)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run
    feed_dict_string, options, run_metadata)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
    target_list, options, run_metadata)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid indices: [272,1] = [0, 15, 832] does not index into [1,832,832,2]
         [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd = ScatterNd[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](ToInt32_3/_2101, adam_vars/gradients/Reshape_4_grad/Reshape/_2103, adam_vars/gradients/GatherNd_1_grad/Shape/_2105)]]
         [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd/_2107 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_155_adam_vars/gradients/GatherNd_1_grad/ScatterNd", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'adam_vars/gradients/GatherNd_1_grad/ScatterNd', defined at:
  File "fcn_32s_train.py", line 100, in <module>
    train_step = tf.train.AdamOptimizer(learning_rate=0.000001).minimize(cross_entropy_sum)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 269, in minimize
    grad_loss=grad_loss)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 335, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 482, in gradients
    in_grads = grad_fn(op, *out_grads)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/array_grad.py", line 354, in _GatherNdGrad
    ref_grad = array_ops.scatter_nd(indices, grad, ref_shape)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2757, in scatter_nd
    shape=shape, name=name)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

...which was originally created as op u'GatherNd_1', defined at:
  File "fcn_32s_train.py", line 82, in <module>
    class_labels=class_labels)
  File "/home/azim_se/projects/laneSegmentation/code/tf_image_segmentation/utils/training.py", line 167, in get_valid_logits_and_labels
    valid_logits_batch_tensor = tf.gather_nd(params=logits_batch_tensor, indices=valid_batch_indices)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1407, in gather_nd
    name=name)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Invalid indices: [272,1] = [0, 15, 832] does not index into [1,832,832,2]
         [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd = ScatterNd[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](ToInt32_3/_2101, adam_vars/gradients/Reshape_4_grad/Reshape/_2103, adam_vars/gradients/GatherNd_1_grad/Shape/_2105)]]
         [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd/_2107 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_155_adam_vars/gradients/GatherNd_1_grad/ScatterNd", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

basically this line shows the error:

InvalidArgumentError (see above for traceback): Invalid indices: [272,1] = [0, 15, 832] does not index into [1,832,832,2]

[warmspringwinds]

Hi, this library will be heavily refactored in a month or so.

So far please check out this: https://github.com/warmspringwinds/dense-ai

It has all the same functionality and even more.

2017-07-24 10:05 GMT-07:00 Seyed Majid Azimi notifications@github.com:

I have changed nothing in running FCN_32s_train.py except changing size and number of classes. image_train_size = [850, 850] number_of_classes = 2 I had created tfrecord successfully, but after running fcn_32s_train.py I get the following error:

Total memory: 11.90GiB Free memory: 11.76GiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:83:00.0) I tensorflow/core/common_runtime/gpu/pool_allocator.cc:247] PoolAllocator: After 44669 get requests, put_count=44143 evicted_count=1000 eviction_rate=0.0226536 and unsatisfied allocation rate=0.0364011 I tensorflow/core/common_runtime/gpu/pool_allocator.cc:259] Raising pool_sizelimit from 100 to 110 W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_shuffle_batch/random_shuffle_queue: Skipping cancelled enqueue attempt with queue not closed Traceback (most recent call last): File "fcn_32s_train.py", line 145, in train_step ]) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run run_metadata_ptr) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run feed_dict_string, options, run_metadata) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run target_list, options, run_metadata) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid indices: [272,1] = [0, 15, 832] does not index into [1,832,832,2] [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd = ScatterNd[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](ToInt32_3/_2101, adam_vars/gradients/Reshape_4_grad/Reshape/_2103, adam_vars/gradients/GatherNd_1_grad/Shape/_2105)]] [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd/_2107 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_155_adam_vars/gradients/GatherNd_1_grad/ScatterNd", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'adam_vars/gradients/GatherNd_1_grad/ScatterNd', defined at: File "fcn_32s_train.py", line 100, in train_step = tf.train.AdamOptimizer(learning_rate=0.000001).minimize(cross_entropy_sum) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 269, in minimize grad_loss=grad_loss) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 335, in compute_gradients colocate_gradients_with_ops=colocate_gradients_with_ops) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py", line 482, in gradients in_grads = grad_fn(op, *out_grads) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/array_grad.py", line 354, in _GatherNdGrad ref_grad = array_ops.scatter_nd(indices, grad, ref_shape) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2757, in scatter_nd shape=shape, name=name) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op op_def=op_def) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op original_op=self._default_original_op, op_def=op_def) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in init self._traceback = _extract_stack()

...which was originally created as op u'GatherNd_1', defined at: File "fcn_32s_train.py", line 82, in class_labels=class_labels) File "/home/azim_se/projects/laneSegmentation/code/tf_image_segmentation/utils/training.py", line 167, in get_valid_logits_and_labels valid_logits_batch_tensor = tf.gather_nd(params=logits_batch_tensor, indices=valid_batch_indices) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1407, in gather_nd name=name) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op op_def=op_def) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op original_op=self._default_original_op, op_def=op_def) File "/home/azim_se/.virtualenvs/dani_sh-tf0.12-py2.7-lane_seg-fcn/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in init self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Invalid indices: [272,1] = [0, 15, 832] does not index into [1,832,832,2] [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd = ScatterNd[T=DT_FLOAT, Tindices=DT_INT32, _device="/job:localhost/ replica:0/task:0/cpu:0"](ToInt32_3/_2101, adam_vars/gradients/Reshape_4_grad/Reshape/_2103, adam_vars/gradients/GatherNd_1_grad/Shape/_2105)]] [[Node: adam_vars/gradients/GatherNd_1_grad/ScatterNd/_2107 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_155adam vars/gradients/GatherNd_1_grad/ScatterNd", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/warmspringwinds/tf-image-segmentation/issues/4#issuecomment-317488590, or mute the thread https://github.com/notifications/unsubscribe-auth/ACYrB0oOQyXBaw7mKGIIsl55QuqMhjHdks5sRM7vgaJpZM4L7kzB .

[arasharchor]

thanks Daniil @warmspringwinds. I am not so familiar with torch. I have worked mostly with tensorflow. That would be super great if you could give me a hint what causes the mentioned error in this library.