problem with "tensorflow.python.framework.errors_impl.OutOfRangeError"

[512792354]

Describe the bug HI, I encounter a bug when testing with command "ae_train exp_group/my_mpencoder ", and it always comes out with the same output like this: "tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found. (0) Out of range: End of sequence". I use 8000 jpg images with resolution of 720540. Also use 3D model in .ply format and convert it into ASCII format with MeshLab software. The model has color. But it finally gets the same problem. Could you help me with that? I attack the whole dialog below. The environment information is also attached. The terminal information: jhl@jhl:~/AugmentedAutoencoder_multipath/a_project$ ae_train exp_group/my_mpencoder -d ('zoom_range: ', [0.8, 1.2]) ('g_noise: ', False) ('contrast_norm_range: ', [0.5, 2.0]) ('mult_brightness: ', [0.6, 1.4]) ('max_off_brightness: ', 0.2) ('gaussian_blur: ', True) ('invert: ', True) WARNING:tensorflow:From /home/jhl/.local/lib/python3.7/site-packages/auto_pose/ae/multi_queue.py:128: DatasetV1.make_initializable_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: This is a deprecated API that should only be used in TF 1 graph mode and legacy TF 2 graph mode available through tf.compat.v1. In all other situations -- namely, eager mode and inside tf.function -- you can consume dataset elements using for elem in dataset: ... or by explicitly creating iterator via iterator = iter(dataset) and fetching its elements via values = next(iterator). Furthermore, this API is not available in TF 2. During the transition from TF 1 to TF 2 you can use tf.compat.v1.data.make_initializable_iterator(dataset) to create a TF 1 graph mode style iterator for a dataset created through TF 2 APIs. Note that this should be a transient state of your code base as there are in general no guarantees about the interoperability of TF 1 and TF 2 code. /home/jhl/AugmentedAutoencoder_multipath/a_project/model/bun_zipper2.ply (None, 128, 128, 3) 0 2 (None, 128, 128, 3) /home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:414: UserWarning: tf.layers.conv2d is deprecated and will be removed in a future version. Please Use tf.keras.layers.Conv2D instead. warnings.warn('tf.layers.conv2d is deprecated and ' /home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1692: UserWarning: layer.apply is deprecated and will be removed in a future version. Please use layer.__call__ method instead. warnings.warn('layer.apply is deprecated and ' /home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/normalization.py:308: UserWarning: tf.layers.batch_normalization is deprecated and will be removed in a future version. Please use tf.keras.layers.BatchNormalization instead. In particular, tf.control_dependencies(tf.GraphKeys.UPDATE_OPS) should not be used (consult the tf.keras.layers.BatchNormalization documentation). 'tf.layers.batch_normalization is deprecated and ' WARNING:tensorflow:From /home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/layers/normalization.py:534: _colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. /home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/core.py:329: UserWarning: tf.layers.flatten is deprecated and will be removed in a future version. Please use tf.keras.layers.Flatten instead. warnings.warn('tf.layers.flatten is deprecated and ' /home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/core.py:171: UserWarning: tf.layers.dense is deprecated and will be removed in a future version. Please use tf.keras.layers.Dense instead. warnings.warn('tf.layers.dense is deprecated and ' 1 (128, 128, 3) [[8, 8], [16, 16], [32, 32], [64, 64]] (None, 128, 128, 3) (None, 128, 128, 3) ('tfrecord exists for', '/home/jhl/AugmentedAutoencoder_multipath/a_project/model/bun_zipper2.ply') 1 2 Tensor("RandomShuffle:0", shape=(2,), dtype=int32) Traceback (most recent call last): File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call return fn(args) File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1360, in _run_fn target_list, run_metadata) File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1453, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found. (0) Out of range: End of sequence [[{{node my_mpencoder/IteratorGetNext_1}}]] [[my_mpencoder/IteratorGetNext_1/_7]] (1) Out of range: End of sequence [[{{node my_mpencoder/IteratorGetNext_1}}]] 0 successful operations. 0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/jhl/.local/bin/ae_train", line 8, in sys.exit(main()) File "/home/jhl/.local/lib/python3.7/site-packages/auto_pose/ae/aetrain.py", line 216, in main this,,reconstr_train = sess.run([multi_queue.next_element,multi_queue.next_bg_element,[decoder.x for decoder in decoders]]) File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 968, in run run_metadata_ptr) File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1191, in _run feed_dict_tensor, options, run_metadata) File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1369, in _do_run run_metadata) File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1394, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found. (0) Out of range: End of sequence [[node my_mpencoder/IteratorGetNext_1 (defined at /lib/python3.7/site-packages/auto_pose/ae/multi_queue.py:174) ]] [[my_mpencoder/IteratorGetNext_1/_7]] (1) Out of range: End of sequence [[node my_mpencoder/IteratorGetNext_1 (defined at /lib/python3.7/site-packages/auto_pose/ae/multi_queue.py:174) ]] 0 successful operations. 0 derived errors ignored.

Errors may have originated from an input operation. Input Source operations connected to node my_mpencoder/IteratorGetNext_1: my_mpencoder/IteratorV2_1 (defined at /lib/python3.7/site-packages/auto_pose/ae/multi_queue.py:172)

Input Source operations connected to node my_mpencoder/IteratorGetNext_1: my_mpencoder/IteratorV2_1 (defined at /lib/python3.7/site-packages/auto_pose/ae/multi_queue.py:172)

Original stack trace for 'my_mpencoder/IteratorGetNext_1': File "/bin/ae_train", line 8, in sys.exit(main()) File "/lib/python3.7/site-packages/auto_pose/ae/ae_train.py", line 102, in main iterator = multi_queue.create_iterator(dataset_path, args) File "/lib/python3.7/site-packages/auto_pose/ae/multi_queue.py", line 174, in create_iterator self.next_element = iterator.get_next() File "/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 420, in get_next name=name) File "/lib/python3.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2750, in iterator_get_next output_shapes=output_shapes, name=name) File "/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 750, in _apply_op_helper attrs=attr_protos, op_def=op_def) File "/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3565, in _create_op_internal op_def=op_def) File "/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2045, in init self._traceback = tf_stack.extract_stack_for_node(self._c_op)

System Info Describe the characteristic of your environment:

• GPU models and configuration:NVIDIA GeForce RTX 2060
• Python version：3.7.6
• OpenGL version:3.1.6
• tensorflow-gpu:2.5.0
• CUDA:11.2

Additional context Add any other context about the problem here.