Parsing the training data

[avnishks]

Hi, First off, thank you for sharing this code. Your paper is really interesting! I am trying to run the code with the JSRT dataset, before I try it on my own. I am using 1024x1024 image-mask pair of JSRT data in png format. I used the whole 247 images as training data and specified the paths as specified in your readme file:

idlist_img_name = os.path.normpath("/home/gpux/JSRT_data/lung_id.txt")
idlist_seg_name = os.path.normpath("/home/gpux/JSRT_data/mask_id.txt")
img_folder_path = os.path.normpath('/home/gpux/JSRT_data/lung')
seg_folder_path = os.path.normpath('/home/gpux/JSRT_data/mask')

But when I train the model, I keep getting the following error: OutOfRangeError (see above for traceback): RandomShuffleQueue '_3_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 16, current size 0)

I am not sure what I am doing wrong here. Do you have any suggestions? (For complete output of the training command please see the attached file: error_msg.txt)

I am using tensorflow 1.4.0 and python 2.7.12+.

[jeya-maria-jose]

Hey , I am also getting the same error when I use my own dataset ! @avnishks Did you resolve it ?

[thomasneff]

Hi!

As far as I can remember, your idlist files should contain only the filenames relative to the base path. (Relative to img_folder_path and seg_folder_path)

That means for the case of @avnishks, lines 44-47 of train.py should probably look like this:

idlist_img_name = 'lung_id.txt' idlist_seg_name = 'mask_id.txt' img_folder_path = '/home/gpux/JSRT_data/lung/' seg_folder_path = '/home/gpux/JSRT_data/mask/'

Your idlists would then probably be identical, and contain lines as follows:

1) lung_id.txt -- START OF FILE JPCNN069.png JPCLN020.png . . . -- END OF FILE

Hope this clears up your confusion, it has been a while since I used this codebase.

It might also be that tensorflow APIs have changed, that I‘m not sure.

[jeya-maria-jose]

Hey , Thanks for the quick response.

Changing it into relative path did not resolve the error .

Getting this error still ::

(tensorflow)jose@jose-Inspiron-3543:~/Generative-Adversarial-Network-based-Synthesis-for-Supervised-Medical-Image-Segmentation$ python train.py 
before d gen

2
4
2048
before d real

2
4
2048
2018-05-15 13:27:38.554389: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Traceback (most recent call last):
  File "train.py", line 154, in <module>
    tf.app.run()
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "train.py", line 150, in main
    train()
  File "train.py", line 85, in train
    _, g_loss_value, d_loss_value = sess.run([train_op, dcgan.losses['g'], dcgan.losses['d']])
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_2_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 0)
     [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

Caused by op u'shuffle_batch', defined at:
  File "train.py", line 154, in <module>
    tf.app.run()
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "train.py", line 150, in main
    train()
  File "train.py", line 54, in train
    images, segmentation_images = load_image_and_segmentation_from_idlist(idlist_tensor_img, idlist_tensor_seg, batch_size, 16, 2560, shift_params = [-128, -0.5], rescale_params = [128, 0.5], shuffle = True)
  File "/home/jose/Generative-Adversarial-Network-based-Synthesis-for-Supervised-Medical-Image-Segmentation/load_folder_images.py", line 84, in load_image_and_segmentation_from_idlist
    min_after_dequeue=min_queue_examples)
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 1225, in shuffle_batch
    name=name)
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 796, in _shuffle_batch
    dequeued = queue.dequeue_many(batch_size, name=name)
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 464, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 2418, in _queue_dequeue_many_v2
    component_types=component_types, timeout_ms=timeout_ms, name=name)
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2957, in create_op
    op_def=op_def)
  File "/home/jose/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1471, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

OutOfRangeError (see above for traceback): RandomShuffleQueue '_2_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 128, current size 0)
     [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, `shuffle_batch/n)]]
``` `

[jeya-maria-jose]

Hey ,

It is working !

adding the txt files inside the respective folders solved the issue.

Thank You :)

[Srijita173]

@jeya-maria-jose I have facing the same problem and it is not getting resolved

For information

idlist_img_name = "/home/nikhitha/Srijita_work/GAN/image_folder/list_of_img_ids.txt" idlist_seg_name = "/home/nikhitha/Srijita_work/GAN/segmentation_folder/list_of_seg_ids.txt" img_folder_path = "/home/nikhitha/Srijita_work/GAN/image_folder" seg_folder_path = "/home/nikhitha/Srijita_work/GAN/segmentation_folder"

[jeya-maria-jose]

Hey @Srijita173 ! I find that you have not included a '/' at the end in your img_folder_path as well as seg_folder_path.