search

michalfaber / tensorflow_Realtime_Multi-Person_Pose_Estimation

Multi-Person Pose Estimation project for Tensorflow 2.0 with a small and fast model based on MobilenetV3
Other
216 stars 65 forks source link

Unknown: RuntimeError: This DataFlow is not reentrant #9

Open rlrahulkanojia opened 4 years ago

rlrahulkanojia commented 4 years ago

This occurs during training after each epoch.

Epoch 4 Step 5280/5304 : Loss paf 20.2253532 Loss heatmap 57.0472374 Total loss 158.943283 Epoch 4 Step 5290/5304 : Loss paf 35.3798218 Loss heatmap 69.6211166 Total loss 213.979401 Epoch 4 Step 5300/5304 : Loss paf 25.4680023 Loss heatmap 53.8446045 Total loss 162.881088 2020-04-01 13:25:00.352075: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled Completed epoch 4. Saving weights... Calculating validation losses... Validation step 0 ... Validation losses for epoch: 4 : Loss paf 63.838680267333984, Loss heatmap 25.57462501525879, Total loss 184.18460083007812 Start processing epoch 5 2020-04-01 13:25:44.597267: W tensorflow/core/framework/op_kernel.cc:1643] Unknown: RuntimeError: This DataFlow is not reentrant! Traceback (most recent call last):

File "/home/unreal/anaconda3/envs/rahul_2.0/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 236, in call ret = func(*args)

File "/home/unreal/anaconda3/envs/rahul_2.0/lib/python3.6/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 789, in generator_py_func values = next(generator_state.get_iterator(iterator_id))

File "/home/unreal/rahul/Documents/pose_tf/dataset/generators.py", line 73, in f for i in df:

File "/home/unreal/anaconda3/envs/rahul_2.0/lib/python3.6/site-packages/tensorpack/dataflow/parallel_map.py", line 309, in iter with self._guard, _zmq_catch_error(type(self).name):

File "/home/unreal/anaconda3/envs/rahul_2.0/lib/python3.6/site-packages/tensorpack/dataflow/base.py", line 36, in enter raise threading.ThreadError("This DataFlow is not reentrant!")

RuntimeError: This DataFlow is not reentrant!