search

shreyashampali / HOnnotate

CVPR2020. HOnnotate: A method for 3D Annotation of Hand and Object Poses
https://www.tugraz.at/index.php?id=40231
GNU General Public License v3.0
166 stars 15 forks source link

OutOfRangeError exception running objectTrackingSingleFrame.py. Intentional? #14

Open petrock99 opened 3 years ago

petrock99 commented 3 years ago

Hello, when I run

python objectTrackingSingleFrame.py --seq 'test'

for '1.1. Object pose initialization' from the ReadMe I get an OutOfRangeError exception at what appears to be the very end of the run:.

maskPC for Image test/0/02294 is 0.036168
[Loading New frame ][0/02294]
'runOptimization'  63.31 ms
0.0034784062
0.0529846
0.026492303
'runOptimization'  561.64 ms
maskPC for Image test/0/02297 is 0.035820
[Loading New frame ][0/02297]
'runOptimization'  78.96 ms
0.003642647
0.060919605
0.03045981
'runOptimization'  1479.31 ms
Traceback (most recent call last):
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
     [[{{node cond/IteratorGetNext}}]]
     [[cond/IteratorGetNext/_15]]
  (1) Out of range: End of sequence
     [[{{node cond/IteratorGetNext}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "objectTrackingSingleFrame.py", line 329, in <module>
    objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
  File "objectTrackingSingleFrame.py", line 142, in objectTracker
    opti1.runOptimization(session, 1, {loadData:True})
  File ".../HOnnotate/HOnnotate/optimization/ghope/utils.py", line 159, in timed
    result = method(*args, **kw)
  File ".../HOnnotate/HOnnotate/optimization/ghope/optimization.py", line 69, in runOptimization
    session.run(self.optOp, feed_dict=feedDict)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
     [[node cond/IteratorGetNext (defined at .../HOnnotate/HOnnotate/optimization/ghope/loss.py:227) ]]
     [[cond/IteratorGetNext/_15]]
  (1) Out of range: End of sequence
     [[node cond/IteratorGetNext (defined at .../HOnnotate/HOnnotate/optimization/ghope/loss.py:227) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'cond/IteratorGetNext':
  File "objectTrackingSingleFrame.py", line 329, in <module>
    objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
  File "objectTrackingSingleFrame.py", line 62, in objectTracker
    frameCntInt, loadData, realObservs = LossObservs.getRealObservables(ds, numFrames, w, h)
  File ".../HOnnotate/HOnnotate/optimization/ghope/loss.py", line 242, in getRealObservables
    lambda: dummyFunc(fidV, segV, depthV, colV, maskV, frameCntIntV))
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1977, in cond
    orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1814, in BuildCondBranch
    original_result = fn()
  File ".../HOnnotate/HOnnotate/optimization/ghope/loss.py", line 241, in <lambda>
    frameID, seg, depth, col, mask, frameCntInt = tf.cond(loadRealObservs, lambda: loadVars(fidV, segV, depthV, colV, maskV, frameCntIntV),
  File ".../HOnnotate/HOnnotate/optimization/ghope/loss.py", line 227, in loadVars
    frameID, seg, depth, col, mask = dataset.make_one_shot_iterator().get_next()
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 426, in get_next
    output_shapes=self._structure._flat_shapes, name=name)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 1947, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File ".../.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

I'm wondering if this is expected/intended as the way for the script to bail out of the while(True) loop in objectTracker(...) in objectTrackingSingleFrame.py? If I add a try/except around session.run(self.optOp, feed_dict=feedDict) in runOptimization(...) the script never ends (due to the while(True) in objectTracker(...).

                try:
                    session.run(self.optOp, feed_dict=feedDict)
                except tf.errors.OutOfRangeError:
                    break # break the while(True)

Should I instead, put the try/except around the call site for objectTracker so that the while(True) will bail out? Is there a better method? Thanks.

                try:
                    objectTracker(w, h, paramInit, camProp, mesh, out_dir, configData)
                except tf.errors.OutOfRangeError:
                    pass    # We're done

Thanks a lot.

Python: 3.6.12 HOnnotate git hash: d94f6b7e1c09a8c7229799a0d48fdc5f2b416780 dirt git hash: 571addc359201b668d9dc450086c6dce6c18d0b6 CUDA: 11 tensorflow: 1.14 GCC: 8.3.1

petrock99 commented 3 years ago

btw, the same problem happens in handObjectTrack in handObjectTrackingSingleFrame.py with the runOptimization on line 190. Frustrating to say the least since it took almost an hour to get to that point.

[Loading New frame ][02298]
'runOptimization'  82.18 ms
'runOptimization'  1429.59 ms
Traceback (most recent call last):
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
     [[{{node cond/IteratorGetNext}}]]
  (1) Out of range: End of sequence
     [[{{node cond/IteratorGetNext}}]]
     [[cond/IteratorGetNext/_19]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "handObjectTrackingSingleFrame.py", line 488, in <module>
    handObjectTrack(w, h, objParamInitList[0], handParamInitList[0], mesh, camProp, out_dir)
  File "handObjectTrackingSingleFrame.py", line 190, in handObjectTrack
    opti1.runOptimization(session, 1, {loadData: True})
  File "/s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/utils.py", line 159, in timed
    result = method(*args, **kw)
  File "/s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/optimization.py", line 68, in runOptimization
    session.run(self.optOp, feed_dict=feedDict)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
     [[node cond/IteratorGetNext (defined at /s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/loss.py:225) ]]
  (1) Out of range: End of sequence
     [[node cond/IteratorGetNext (defined at /s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/loss.py:225) ]]
     [[cond/IteratorGetNext/_19]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'cond/IteratorGetNext':
  File "handObjectTrackingSingleFrame.py", line 488, in <module>
    handObjectTrack(w, h, objParamInitList[0], handParamInitList[0], mesh, camProp, out_dir)
  File "handObjectTrackingSingleFrame.py", line 65, in handObjectTrack
    frameCntInt, loadData, realObservs = LossObservs.getRealObservables(ds, numFrames, w, h)
  File "/s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/loss.py", line 240, in getRealObservables
    lambda: dummyFunc(fidV, segV, depthV, colV, maskV, frameCntIntV))
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1977, in cond
    orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1814, in BuildCondBranch
    original_result = fn()
  File "/s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/loss.py", line 239, in <lambda>
    frameID, seg, depth, col, mask, frameCntInt = tf.cond(loadRealObservs, lambda: loadVars(fidV, segV, depthV, colV, maskV, frameCntIntV),
  File "/s/parsons/h/proj/vision/usr/petrock/HOnnotate/HOnnotate/optimization/ghope/loss.py", line 225, in loadVars
    frameID, seg, depth, col, mask = dataset.make_one_shot_iterator().get_next()
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 426, in get_next
    output_shapes=self._structure._flat_shapes, name=name)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 1947, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/s/chopin/l/grad/petrock/.conda/envs/HOnnotate/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()
shreyashampali commented 3 years ago

Hi, Sorry for the late reply here. You are right, the tracking scripts do not have a graceful exit at the sequence and the problem is mainly with the while(true) loop. Irrespective of whether the script crashes or not at the end, you should have the poses dumped in the output folders which are required for subsequent stages. One way to fix the crash at the end of the sequence is to replace the while loop with a for loop if you know the number of frames you are tracking for.