OutOfRangeError:

[xjtushujun]

It appears that:

Traceback (most recent call last): File "main.py", line 110, in main() File "main.py", line 106, in main selfie(gpu_id, input_reader, model_name, total_epochs, batch_size, lr_boundaries, lr_values, optimizer, noise_rate, noise_type, warm_up, threshold, queue_size, restart=restart, log_dir=log_dir) File "/media/shujun/sj/project/SELFIE/SELFIE/algorithm/selfie.py", line 154, in selfie train_batch_patcher.bulk_load_in_memory(sess, train_ids, train_images, train_labels) File "/media/shujun/sj/project/SELFIE/SELFIE/reader/batch_patcher.py", line 40, in bulk_load_in_memory mini_ids, mini_images, mini_labels = sess.run([ids, images, labels]) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 900, in run run_metadata_ptr) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_0_shuffle_batch/fifo_queue' is closed and has insufficient elements (requested 128, current size 0) [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_UINT8, DT_FLOAT, DT_UINT8], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/fifo_queue, shuffle_batch/n/_147)]]

[songhwanjun]

I think the problem was caused by the wrong path of input directory. Please check it and let me know. Thanks.

[xjtushujun]

Now read following files. ['/media/shujun/sj/project/SELFIE/SELFIE/dataset/CIFAR-10/data_batch_1.bin', '/media/shujun/sj/project/SELFIE/SELFIE/dataset/CIFAR-10/data_batch_2.bin', '/media/shujun/sj/project/SELFIE/SELFIE/dataset/CIFAR-10/data_batch_3.bin', '/media/shujun/sj/project/SELFIE/SELFIE/dataset/CIFAR-10/data_batch_4.bin', '/media/shujun/sj/project/SELFIE/SELFIE/dataset/CIFAR-10/data_batch_5.bin'] Filling queue with 20000 data before starting to train. This will take a few minutes. Now read following files. ['/media/shujun/sj/project/SELFIE/SELFIE/dataset/CIFAR-10/test_batch.bin']

It seems to read the data correctly.

[songhwanjun]

When I tried to run again, it was work well as follow logs:

Now read following files. ['/data/home/songhwanjun/testing/SELFIE/dataset/CIFAR-10/data_batch_1.bin', '/data/home/songhwanjun/testing/SELFIE/dataset/CIFAR-10/data_batch_2.bin', '/data/home/songhwanjun/testing/SELFIE/dataset/CIFAR-10/data_batch_3.bin', '/data/home/songhwanjun/testing/SELFIE/dataset/CIFAR-10/data_batch_4.bin', '/data/home/songhwanjun/testing/SELFIE/dataset/CIFAR-10/data_batch_5.bin'] Filling queue with 20000 data before starting to train. This will take a few minutes. Now read following files. ['/data/home/songhwanjun/testing/SELFIE/dataset/CIFAR-10/test_batch.bin'] Filling queue with 4000 data before starting to train. This will take a few minutes. [0731 12:50:52 @registry.py:121] DenseNet/conv0 input: [None, 32, 32, 3] [0731 12:50:52 @registry.py:129] DenseNet/conv0 output: [None, 32, 32, 16] [0731 12:50:52 @registry.py:121] DenseNet/block1/dense_layer.0/conv1 input: [None, 32, 32, 16] [0731 12:50:52 @registry.py:129] DenseNet/block1/dense_layer.0/conv1 output: [None, 32, 32, 12] [0731 12:50:52 @registry.py:121] DenseNet/block1/dense_layer.1/conv1 input: [None, 32, 32, 28] [0731 12:50:52 @registry.py:129] DenseNet/block1/dense_layer.1/conv1 output: [None, 32, 32, 12] [0731 12:50:52 @registry.py:121] DenseNet/block1/dense_layer.2/conv1 input: [None, 32, 32, 40] [0731 12:50:52 @registry.py:129] DenseNet/block1/dense_layer.2/conv1 output: [None, 32, 32, 12] [0731 12:50:52 @registry.py:121] DenseNet/block1/dense_layer.3/conv1 input: [None, 32, 32, 52] [0731 12:50:52 @registry.py:129] DenseNet/block1/dense_layer.3/conv1 output: [None, 32, 32, 12] [0731 12:50:52 @registry.py:121] DenseNet/block1/dense_layer.4/conv1 input: [None, 32, 32, 64] [0731 12:50:52 @registry.py:129] DenseNet/block1/dense_layer.4/conv1 output: [None, 32, 32, 12] [0731 12:50:53 @registry.py:121] DenseNet/block1/dense_layer.5/conv1 input: [None, 32, 32, 76] [0731 12:50:53 @registry.py:129] DenseNet/block1/dense_layer.5/conv1 output: [None, 32, 32, 12] [0731 12:50:53 @registry.py:121] DenseNet/block1/dense_layer.6/conv1 input: [None, 32, 32, 88] [0731 12:50:53 @registry.py:129] DenseNet/block1/dense_layer.6/conv1 output: [None, 32, 32, 12] [0731 12:50:53 @registry.py:121] DenseNet/block1/transition1/conv1 input: [None, 32, 32, 100] [0731 12:50:53 @registry.py:129] DenseNet/block1/transition1/conv1 output: [None, 32, 32, 100] [0731 12:50:53 @registry.py:121] DenseNet/block1/transition1/pool input: [None, 32, 32, 100] [0731 12:50:53 @registry.py:129] DenseNet/block1/transition1/pool output: [None, 16, 16, 100] [0731 12:50:53 @registry.py:121] DenseNet/block2/dense_layer.0/conv1 input: [None, 16, 16, 100] [0731 12:50:53 @registry.py:129] DenseNet/block2/dense_layer.0/conv1 output: [None, 16, 16, 12] [0731 12:50:53 @registry.py:121] DenseNet/block2/dense_layer.1/conv1 input: [None, 16, 16, 112] [0731 12:50:53 @registry.py:129] DenseNet/block2/dense_layer.1/conv1 output: [None, 16, 16, 12] [0731 12:50:53 @registry.py:121] DenseNet/block2/dense_layer.2/conv1 input: [None, 16, 16, 124] [0731 12:50:53 @registry.py:129] DenseNet/block2/dense_layer.2/conv1 output: [None, 16, 16, 12] [0731 12:50:53 @registry.py:121] DenseNet/block2/dense_layer.3/conv1 input: [None, 16, 16, 136] [0731 12:50:53 @registry.py:129] DenseNet/block2/dense_layer.3/conv1 output: [None, 16, 16, 12] [0731 12:50:53 @registry.py:121] DenseNet/block2/dense_layer.4/conv1 input: [None, 16, 16, 148] [0731 12:50:53 @registry.py:129] DenseNet/block2/dense_layer.4/conv1 output: [None, 16, 16, 12] [0731 12:50:53 @registry.py:121] DenseNet/block2/dense_layer.5/conv1 input: [None, 16, 16, 160] [0731 12:50:53 @registry.py:129] DenseNet/block2/dense_layer.5/conv1 output: [None, 16, 16, 12] [0731 12:50:53 @registry.py:121] DenseNet/block2/dense_layer.6/conv1 input: [None, 16, 16, 172] [0731 12:50:53 @registry.py:129] DenseNet/block2/dense_layer.6/conv1 output: [None, 16, 16, 12] [0731 12:50:53 @registry.py:121] DenseNet/block2/transition2/conv1 input: [None, 16, 16, 184] [0731 12:50:53 @registry.py:129] DenseNet/block2/transition2/conv1 output: [None, 16, 16, 184] [0731 12:50:53 @registry.py:121] DenseNet/block2/transition2/pool input: [None, 16, 16, 184] [0731 12:50:53 @registry.py:129] DenseNet/block2/transition2/pool output: [None, 8, 8, 184] [0731 12:50:53 @registry.py:121] DenseNet/block3/dense_layer.0/conv1 input: [None, 8, 8, 184] [0731 12:50:53 @registry.py:129] DenseNet/block3/dense_layer.0/conv1 output: [None, 8, 8, 12] [0731 12:50:53 @registry.py:121] DenseNet/block3/dense_layer.1/conv1 input: [None, 8, 8, 196] [0731 12:50:53 @registry.py:129] DenseNet/block3/dense_layer.1/conv1 output: [None, 8, 8, 12] [0731 12:50:53 @registry.py:121] DenseNet/block3/dense_layer.2/conv1 input: [None, 8, 8, 208] [0731 12:50:53 @registry.py:129] DenseNet/block3/dense_layer.2/conv1 output: [None, 8, 8, 12] [0731 12:50:53 @registry.py:121] DenseNet/block3/dense_layer.3/conv1 input: [None, 8, 8, 220] [0731 12:50:53 @registry.py:129] DenseNet/block3/dense_layer.3/conv1 output: [None, 8, 8, 12] [0731 12:50:53 @registry.py:121] DenseNet/block3/dense_layer.4/conv1 input: [None, 8, 8, 232] [0731 12:50:53 @registry.py:129] DenseNet/block3/dense_layer.4/conv1 output: [None, 8, 8, 12] [0731 12:50:53 @registry.py:121] DenseNet/block3/dense_layer.5/conv1 input: [None, 8, 8, 244] [0731 12:50:53 @registry.py:129] DenseNet/block3/dense_layer.5/conv1 output: [None, 8, 8, 12] [0731 12:50:53 @registry.py:121] DenseNet/block3/dense_layer.6/conv1 input: [None, 8, 8, 256] [0731 12:50:53 @registry.py:129] DenseNet/block3/dense_layer.6/conv1 output: [None, 8, 8, 12] [0731 12:50:53 @registry.py:121] DenseNet/gap input: [None, 8, 8, 268] [0731 12:50:53 @registry.py:129] DenseNet/gap output: [None, 268] [0731 12:50:53 @registry.py:121] DenseNet/linear input: [None, 268] [0731 12:50:53 @registry.py:129] DenseNet/linear output: [None, 10] '# of samples: 50000' '# of samples: 10000' Noise Injection: pair history length : 15 Restart: 0 1 , 0.1 , 3.9554359089687963 , 0.2579323849104859 , 1.7421533110775524 , 0.3686708860759494

I think that CIFAR-10 data did not exist in your path: '/media/shujun/sj/project/SELFIE/SELFIE/dataset/CIFAR-10/'. Please download "data_batch_x.bin and test_batch.bin and locate it inside the path. (you can download the dataset by clicking the link in my github.)