InvalidArgumentError: logits and labels must be same size

[yadmitri]

Hello,

i am trying to run tf_unet implementation on 16bit tiff images with 4 bands. i have already checked all related threads here around tf_unet repo:

• created CustomImageDataProvider with ability to read 16bit tiffs
• cleaned training data - every tiff contains two training classes always
• checked shapes of test_x, test_y = data_provider(4) and results are (4, 75, 75, 4) and (4, 75, 75, 2)

The issue InvalidArgumentError: logits and labels must be same size: logits_size=[4096,2] labels_size=[4356,2] is still exists.

Could anyone please advice next steps that i could try or could look at? Some more code details is below. Many thanks in advance.

Code snippet:

from tf_unet import unet, util, image_util, tifffile
from PIL import Image

class CustomImageDataProvider(image_util.ImageDataProvider):
    def _load_file(self, path, dtype=np.float32):
         return tifffile.imread(path)

data_provider = CustomImageDataProvider(search_path=LOCAL_PREPARED_16BIT_TRAINING_DIR + "/*.tiff", data_suffix="_16bit.tiff", mask_suffix="_mask.tiff", n_class=2)

test_x, test_y = data_provider(4)
print(test_x.shape)
print(test_y.shape)

net = unet.Unet(channels=data_provider.channels, n_class=data_provider.n_class, layers=3, features_root=64)

trainer = unet.Trainer(net, batch_size=20, optimizer="momentum", opt_kwargs=dict(momentum=0.2))

path = trainer.train(data_provider, "unet_trained", training_iters=10, epochs=100, display_step=5, dropout=0.75)

Log:

InvalidArgumentError: logits and labels must be same size: logits_size=[4096,2] labels_size=[4356,2]
     [[Node: softmax_cross_entropy_with_logits_sg = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](softmax_cross_entropy_with_logits_sg/Reshape, softmax_cross_entropy_with_logits_sg/Reshape_1)]]

During handling of the above exception, another exception occurred:

InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-40-5147d68ae806> in <module>()
      7                      epochs=100,
      8                      display_step=5,
----> 9                      dropout=0.75)

~/.local/lib/python3.6/site-packages/tf_unet-0.1.1-py3.6.egg/tf_unet/unet.py in train(self, data_provider, output_path, training_iters, epochs, dropout, display_step, restore, write_graph, prediction_path)
    407 
    408             test_x, test_y = data_provider(self.verification_batch_size)
--> 409             pred_shape = self.store_prediction(sess, test_x, test_y, "_init")
    410 
    411             summary_writer = tf.summary.FileWriter(output_path, graph=sess.graph)

~/.local/lib/python3.6/site-packages/tf_unet-0.1.1-py3.6.egg/tf_unet/unet.py in store_prediction(self, sess, batch_x, batch_y, name)
    450         loss = sess.run(self.net.cost, feed_dict={self.net.x: batch_x, 
    451                                                        self.net.y: util.crop_to_shape(batch_y, pred_shape),
--> 452                                                        self.net.keep_prob: 1.})
    453 
    454         logging.info("Verification error= {:.1f}%, loss= {:.4f}".format(error_rate(prediction,

Best regards, Dzmitry

[SUYEgit]

Hi, I guess you might have different number of images and mask within one folder. That should be the reason this mismatch happens.

logits_size=[4096,2] labels_size=[4356,2]

Sincerely, Su Ye

National University of Singapore|Harbin Institute of Technology

Tel. +65 98115724

emailto:arjun@tum-create.edu.sg0178254@u.nus.edu

---

发件人: yadmitri notifications@github.com 发送时间: 2018年4月2日 5:20:25 收件人: jakeret/tf_unet 抄送: Subscribed 主题: [jakeret/tf_unet] InvalidArgumentError: logits and labels must be same size (#165)

Hello,

i am trying to run tf_unet implementation on 16bit tiff images with 4 bands. i have already checked all related threads here around tf_unet repo:

• created CustomImageDataProvider with ability to read 16bit tiffs
• cleaned training data - every tiff contains two training classes always
• checked shapes of test_x, test_y = data_provider(4) and results are (4, 75, 75, 4) and (4, 75, 75, 2)

The issue InvalidArgumentError: logits and labels must be same size: logits_size=[4096,2] labels_size=[4356,2] is still exists.

Could anyone please advice next steps that i could try or could look at? Some more code details is below. Many thanks in advance.

Code snippet:

from tf_unet import unet, util, image_util, tifffile from PIL import Image

class CustomImageDataProvider(image_util.ImageDataProvider): def _load_file(self, path, dtype=np.float32): return tifffile.imread(path)

data_provider = CustomImageDataProvider(search_path=LOCAL_PREPARED_16BIT_TRAINING_DIR + "/*.tiff", data_suffix="_16bit.tiff", mask_suffix="_mask.tiff", n_class=2)

test_x, test_y = data_provider(4) print(test_x.shape) print(test_y.shape)

net = unet.Unet(channels=data_provider.channels, n_class=data_provider.n_class, layers=3, features_root=64)

trainer = unet.Trainer(net, batch_size=20, optimizer="momentum", opt_kwargs=dict(momentum=0.2))

path = trainer.train(data_provider, "unet_trained", training_iters=10, epochs=100, display_step=5, dropout=0.75)

Log:

InvalidArgumentError: logits and labels must be same size: logits_size=[4096,2] labels_size=[4356,2] [[Node: softmax_cross_entropy_with_logits_sg = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](softmax_cross_entropy_with_logits_sg/Reshape, softmax_cross_entropy_with_logits_sg/Reshape_1)]]

During handling of the above exception, another exception occurred:

InvalidArgumentError Traceback (most recent call last)

in () 7 epochs=100, 8 display_step=5, ----> 9 dropout=0.75) ~/.local/lib/python3.6/site-packages/tf_unet-0.1.1-py3.6.egg/tf_unet/unet.py in train(self, data_provider, output_path, training_iters, epochs, dropout, display_step, restore, write_graph, prediction_path) 407 408 test_x, test_y = data_provider(self.verification_batch_size) --> 409 pred_shape = self.store_prediction(sess, test_x, test_y, "_init") 410 411 summary_writer = tf.summary.FileWriter(output_path, graph=sess.graph) ~/.local/lib/python3.6/site-packages/tf_unet-0.1.1-py3.6.egg/tf_unet/unet.py in store_prediction(self, sess, batch_x, batch_y, name) 450 loss = sess.run(self.net.cost, feed_dict={self.net.x: batch_x, 451 self.net.y: util.crop_to_shape(batch_y, pred_shape), --> 452 self.net.keep_prob: 1.}) 453 454 logging.info("Verification error= {:.1f}%, loss= {:.4f}".format(error_rate(prediction, Best regards, Dzmitry ― You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[yadmitri]

Hi Su Ye,

My checks:

1. Every mask tiff has corresponding image tiff.
2. Runing similar training on subset (5% - 67 pairs of tiffs) produces the similar error with similar numbers InvalidArgumentError: logits and labels must be same size: logits_size=[4096,2] labels_size=[4356,2]

Any more ideas ?

Best regards, Dzmitry

[xljsh]

@SUYENus I meet the similar problem. Can you give me some advice? the problem link: https://stackoverflow.com/questions/49604062/logits-and-labels-must-be-same-size-logits-size-64-2-labels-size-32-2

[SUYEgit]

Hi,

As I know there is a' crop_to_shape' function in the code which ensures the logits from network has the same shape with labels. Did you do any modification on that?

Sincerely, Su Ye

National University of Singapore|Harbin Institute of Technology

Tel. +65 98115724

emailto:arjun@tum-create.edu.sg0178254@u.nus.edu

---

发件人: yadmitri notifications@github.com 发送时间: 2018年4月2日 17:55:11 收件人: jakeret/tf_unet 抄送: Su Ye; Comment 主题: Re: [jakeret/tf_unet] InvalidArgumentError: logits and labels must be same size (#165)

Hi Su Ye,

My checks:

1. Every mask tiff has corresponding image tiff.
2. Runing similar training on subset (5% - 67 pairs of tiffs) produces the similar error with similar numbers InvalidArgumentError: logits and labels must be same size: logits_size=[4096,2] labels_size=[4356,2]

Any more ideas ?

Best regards, Dzmitry

― You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/jakeret/tf_unet/issues/165#issuecomment-377897944, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AhnUCYQ-XUQ97RyNE7Qx2U5RBgAiUP2aks5tkfV_gaJpZM4TC-du.

[yadmitri]

@SUYENus , No changes in core library jakeret/tf_unet. I only added CustomImageDataProvider to read 16bit tiffs (already mentioned in parent issue description).

Thanks, Dzmitry

[yadmitri]

Original issue was in pixel size of training data. 75x75 image size did not fit good at all, so resizing to 32x32 resolved the issue.

The root issue was in crop_to_shape and dividing by 2:

def crop_to_shape(data, shape):
    offset0 = (data.shape[1] - shape[1])//2
    offset1 = (data.shape[2] - shape[2])//2
    return data[:, offset0:(-offset0), offset1:(-offset1)]

Best regards, Dzmitry