search

MaybeShewill-CV / CRNN_Tensorflow

Convolutional Recurrent Neural Networks(CRNN) for Scene Text Recognition
MIT License
1.03k stars 388 forks source link

Val generator get next error #423

Closed YanaHontarenko closed 3 years ago

YanaHontarenko commented 3 years ago

Hello! I try to train model on previously created tfrecords.

Error which I get:

I1203 18:27:07.126662 30607 train_shadownet.py:573] Use single gpu to train the model
WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.parse_example is deprecated. Please use tf.io.parse_example instead.

W1203 18:27:07.246248 30607 module_wrapper.py:139] From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.parse_example is deprecated. Please use tf.io.parse_example instead.

WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W1203 18:27:07.246532 30607 module_wrapper.py:139] From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

W1203 18:27:07.246757 30607 module_wrapper.py:139] From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/data_provider/tf_io_pipline_fast_tools.py:404: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
W1203 18:27:07.428228 30607 deprecation.py:323] From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/data_provider/tf_io_pipline_fast_tools.py:404: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.
WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/crnn_net.py:219: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W1203 18:27:07.510576 30607 module_wrapper.py:139] From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/crnn_net.py:219: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

W1203 18:27:07.510931 30607 lazy_loader.py:50] 
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/cnn_basenet.py:73: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W1203 18:27:07.511185 30607 module_wrapper.py:139] From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/cnn_basenet.py:73: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/cnn_basenet.py:332: batch_normalization (from tensorflow.python.layers.normalization) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.batch_normalization` documentation).
W1203 18:27:07.520884 30607 deprecation.py:323] From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/cnn_basenet.py:332: batch_normalization (from tensorflow.python.layers.normalization) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.BatchNormalization instead.  In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.batch_normalization` documentation).
WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/layers/normalization.py:327: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
W1203 18:27:07.521735 30607 deprecation.py:323] From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/layers/normalization.py:327: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/cnn_basenet.py:145: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

W1203 18:27:07.540522 30607 module_wrapper.py:139] From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/cnn_basenet.py:145: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/crnn_net.py:171: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
W1203 18:27:07.692416 30607 deprecation.py:323] From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/crnn_net.py:171: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/contrib/rnn/python/ops/rnn.py:239: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API
W1203 18:27:07.693273 30607 deprecation.py:323] From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/contrib/rnn/python/ops/rnn.py:239: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API
WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/rnn.py:464: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell)`, which is equivalent to this API
W1203 18:27:07.693452 30607 deprecation.py:323] From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/rnn.py:464: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell)`, which is equivalent to this API
WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/rnn_cell_impl.py:958: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
W1203 18:27:07.730970 30607 deprecation.py:323] From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/rnn_cell_impl.py:958: Layer.add_variable (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.add_weight` method instead.
WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/rnn_cell_impl.py:962: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W1203 18:27:07.736938 30607 deprecation.py:506] From /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/rnn_cell_impl.py:962: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/cnn_basenet.py:280: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
W1203 18:27:07.963938 30607 deprecation.py:506] From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/cnn_basenet.py:280: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/crnn_net.py:251: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

W1203 18:27:07.992928 30607 module_wrapper.py:139] From /home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/crnn_model/crnn_net.py:251: The name tf.nn.ctc_loss is deprecated. Please use tf.compat.v1.nn.ctc_loss instead.

WARNING:tensorflow:From tools/train_shadownet.py:219: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.

W1203 18:27:08.297156 30607 module_wrapper.py:139] From tools/train_shadownet.py:219: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.

WARNING:tensorflow:From tools/train_shadownet.py:227: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

W1203 18:27:08.302697 30607 module_wrapper.py:139] From tools/train_shadownet.py:227: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From tools/train_shadownet.py:227: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

W1203 18:27:08.302831 30607 module_wrapper.py:139] From tools/train_shadownet.py:227: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From tools/train_shadownet.py:229: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.

W1203 18:27:08.303001 30607 module_wrapper.py:139] From tools/train_shadownet.py:229: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.

WARNING:tensorflow:From tools/train_shadownet.py:236: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W1203 18:27:09.545008 30607 module_wrapper.py:139] From tools/train_shadownet.py:236: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From tools/train_shadownet.py:244: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

W1203 18:27:09.547039 30607 module_wrapper.py:139] From tools/train_shadownet.py:244: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From tools/train_shadownet.py:247: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W1203 18:27:09.547549 30607 module_wrapper.py:139] From tools/train_shadownet.py:247: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From tools/train_shadownet.py:255: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W1203 18:27:09.647450 30607 module_wrapper.py:139] From tools/train_shadownet.py:255: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From tools/train_shadownet.py:259: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

W1203 18:27:09.647606 30607 module_wrapper.py:139] From tools/train_shadownet.py:259: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-12-03 18:27:09.647891: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-12-03 18:27:09.670855: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2799925000 Hz
2020-12-03 18:27:09.671226: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x510be60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-03 18:27:09.671242: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-12-03 18:27:09.672865: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-12-03 18:27:10.107710: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-03 18:27:10.108093: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x507b9a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-12-03 18:27:10.108109: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 2070, Compute Capability 7.5
2020-12-03 18:27:10.108249: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-03 18:27:10.108538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce RTX 2070 major: 7 minor: 5 memoryClockRate(GHz): 1.74
pciBusID: 0000:01:00.0
2020-12-03 18:27:10.108694: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-12-03 18:27:10.109526: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-12-03 18:27:10.110257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-12-03 18:27:10.110431: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-12-03 18:27:10.111412: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-12-03 18:27:10.112163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-12-03 18:27:10.114618: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-12-03 18:27:10.115005: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-03 18:27:10.115437: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-03 18:27:10.115717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-12-03 18:27:10.115766: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-12-03 18:27:10.116550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-03 18:27:10.116563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2020-12-03 18:27:10.116568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2020-12-03 18:27:10.116746: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-03 18:27:10.117168: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-03 18:27:10.117656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7184 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:From tools/train_shadownet.py:261: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W1203 18:27:10.118476 30607 module_wrapper.py:139] From tools/train_shadownet.py:261: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

I1203 18:27:10.285004 30607 train_shadownet.py:270] Training from scratch
WARNING:tensorflow:From tools/train_shadownet.py:271: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

W1203 18:27:10.285955 30607 module_wrapper.py:139] From tools/train_shadownet.py:271: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

2020-12-03 18:27:11.930089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
Traceback (most recent call last):
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
         [[{{node val_IteratorGetNext}}]]
         [[CTCLoss/_81]]
  (1) Out of range: End of sequence
         [[{{node val_IteratorGetNext}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tools/train_shadownet.py", line 579, in <module>
    need_decode=args.decode_outputs
  File "tools/train_shadownet.py", line 323, in train_shadownet
    [optimizer, train_ctc_loss, merge_summary_op])
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
         [[node val_IteratorGetNext (defined at /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
         [[CTCLoss/_81]]
  (1) Out of range: End of sequence
         [[node val_IteratorGetNext (defined at /home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'val_IteratorGetNext':
  File "tools/train_shadownet.py", line 579, in <module>
    need_decode=args.decode_outputs
  File "tools/train_shadownet.py", line 160, in train_shadownet
    batch_size=CFG.TRAIN.BATCH_SIZE
  File "/home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/data_provider/shadownet_data_feed_pipline.py", line 289, in inputs
    num_threads=CFG.TRAIN.CPU_MULTI_PROCESS_NUMS
  File "/home/yhontarenko/PycharmProjects/text-recognition/CRNN_Tensorflow/data_provider/tf_io_pipline_fast_tools.py", line 406, in inputs
    return iterator.get_next(name='{:s}_IteratorGetNext'.format(self._dataset_flag))
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next
    name=name)gpu GeForce RTX 2070
tensorflow-gpu 1.15
CUDA 10.0
cuDNN 7.6.5
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/home/yhontarenko/PycharmProjects/text-recognition/venv/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

gpu GeForce RTX 2070 tensorflow-gpu 1.15 CUDA 10.0 cuDNN 7.6.5

MaybeShewill-CV commented 3 years ago

@hontarenkoYana Check if your tfrecords was successfully generated:)

YanaHontarenko commented 3 years ago

@MaybeShewill-CV how can I check this?

MaybeShewill-CV commented 3 years ago

@hontarenkoYana Decode tfrecords to see if you can recover your source data. As for the method to decode tfrecords your may refer to tensorflow's io tutorials:)

YanaHontarenko commented 3 years ago

@MaybeShewill-CV , do you have ideas what could goes wrong when tfrecords were creating? I used script which you had provided

MaybeShewill-CV commented 3 years ago

@hontarenkoYana Do not know what happened here. None of this should happen if you follow README:)

YanaHontarenko commented 3 years ago

@MaybeShewill-CV , I found an error! seems my PC hadn't enough space when tfrecords was created and that's why records was broken! Everything is okay now, thank you for your time and work!

MaybeShewill-CV commented 3 years ago

@hontarenkoYana 👍

dcdethan commented 3 years ago

@MaybeShewill-CV , I found an error! seems my PC hadn't enough space when tfrecords was created and that's why records was broken! Everything is okay now, thank you for your time and work!

are you sure?