gpcv-liujin / REDNet

implementation of REDNet (A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-view Stereo Reconstruction from An Open Aerial Dataset)
53 stars 11 forks source link

The graph couldn't be sorted in topological order. #16

Closed NanCheng2001 closed 1 year ago

NanCheng2001 commented 1 year ago

Hello, I used viewselection.py to correctly generate viewpair.txt. Other configurations are the same as yours. I also used the Tensorflow1.13.1 version, and the cuda version is correct However, the following errors occurred when the prediction.py program was run. The errors occurred when the pre training weights were imported

Pre-trained model restored from C:/Users/PC2080ti/Desktop/Mulit-View/jishunping/1/REDNet-master/MODEL_FOLDER/model.ckpt-150000
2022-12-18 18:58:03.486554: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:704] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2022-12-18 18:58:03.493902: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:704] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.
NanCheng2001 commented 1 year ago

More details are as follows:

Pre-trained model restored from C:/Users/PC2080ti/Desktop/Mulit-View/jishunping/1/REDNet-master/MODEL_FOLDER/model.ckpt-150000
2022-12-18 18:58:03.486554: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:704] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2022-12-18 18:58:03.493902: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:704] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.
2022-12-18 19:21:23.300856: E tensorflow/stream_executor/cuda/cuda_blas.cc:698] failed to run cuBLAS routine cublasGemmBatchedEx: CUBLAS_STATUS_EXECUTION_FAILED
2022-12-18 19:21:23.301035: E tensorflow/stream_executor/cuda/cuda_blas.cc:2620] Internal: failed BLAS call, see log for details
Traceback (most recent call last):
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
    return fn(*args)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas xGEMMBatched launch failed : a.shape=[8,3,3], b.shape=[8,3,3], m=3, n=3, k=3, batch_size=8
     [[{{node get_homographies/MatMul_2}}]]
     [[{{node while/RefExit}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/PC2080ti/Desktop/Mulit-View/jishunping/1/REDNet-master/predict.py", line 264, in <module>
    tf.app.run()
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "C:/Users/PC2080ti/Desktop/Mulit-View/jishunping/1/REDNet-master/predict.py", line 259, in main
    rednet_pipeline(mvs_list)
  File "C:/Users/PC2080ti/Desktop/Mulit-View/jishunping/1/REDNet-master/predict.py", line 206, in rednet_pipeline
    [init_depth_map, prob_map, croped_images, scaled_cams, croped_cams, locations])
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
    run_metadata_ptr)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
    run_metadata)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas xGEMMBatched launch failed : a.shape=[8,3,3], b.shape=[8,3,3], m=3, n=3, k=3, batch_size=8
     [[node get_homographies/MatMul_2 (defined at C:\Users\PC2080ti\Desktop\Mulit-View\jishunping\1\REDNet-master\homography_warping.py:45) ]]
     [[node while/RefExit (defined at C:\Users\PC2080ti\Desktop\Mulit-View\jishunping\1\REDNet-master\model.py:294) ]]

Caused by op 'get_homographies/MatMul_2', defined at:
  File "C:/Users/PC2080ti/Desktop/Mulit-View/jishunping/1/REDNet-master/predict.py", line 264, in <module>
    tf.app.run()
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run
    _sys.exit(main(argv))
  File "C:/Users/PC2080ti/Desktop/Mulit-View/jishunping/1/REDNet-master/predict.py", line 259, in main
    rednet_pipeline(mvs_list)
  File "C:/Users/PC2080ti/Desktop/Mulit-View/jishunping/1/REDNet-master/predict.py", line 173, in rednet_pipeline
    depth_num, depth_start, depth_end)
  File "C:\Users\PC2080ti\Desktop\Mulit-View\jishunping\1\REDNet-master\model.py", line 163, in inference_winner_take_all
    depth_start=depth_start, depth_interval=depth_interval)
  File "C:\Users\PC2080ti\Desktop\Mulit-View\jishunping\1\REDNet-master\homography_warping.py", line 45, in get_homographies_Twc
    middle_mat2 = tf.matmul(middle_mat0, middle_mat1)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\ops\math_ops.py", line 2417, in matmul
    a, b, adj_x=adjoint_a, adj_y=adjoint_b, name=name)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1483, in batch_mat_mul
    "BatchMatMul", x=x, y=y, adj_x=adj_x, adj_y=adj_y, name=name)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
    op_def=op_def)
  File "C:\Users\PC2080ti\anaconda3\envs\pyxj37\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

InternalError (see above for traceback): Blas xGEMMBatched launch failed : a.shape=[8,3,3], b.shape=[8,3,3], m=3, n=3, k=3, batch_size=8
     [[node get_homographies/MatMul_2 (defined at C:\Users\PC2080ti\Desktop\Mulit-View\jishunping\1\REDNet-master\homography_warping.py:45) ]]
     [[node while/RefExit (defined at C:\Users\PC2080ti\Desktop\Mulit-View\jishunping\1\REDNet-master\model.py:294) ]]
NanCheng2001 commented 1 year ago

OK, I see. It's my 3090 video card that doesn't fit into cuda10. Excuse me