训练速度过慢

[bonnie-cbw]

求助求助！ 您好，我使用的裁剪为800×800的DOTA数据集，ResNet50，按照您的教程一步步做的，环境的配置也没有什么问题，但是训练速度大概在2.8秒一张图。这个速度是不是有一些太慢了？想问问各位大佬知道有可能是什么原因吗？ 谢谢！！！

[yangxue0827]

没用上gpu吧，cfgs上需要设置一下

[bonnie-cbw]

非常感谢您的解答！ 这是我cfgs中一部分的设置 这个gpu_group我应该有设置呀，是设置的有什么问题吗？ 我现在用的服务器是两块2080Ti的显卡

---------------------------------------- System_config

ROOT_PATH = os.path.abspath('../') print(3*"++--") print(ROOT_PATH) GPU_GROUP = "0,1" NUM_GPU = len(GPU_GROUP.strip().split(',')) SHOW_TRAIN_INFO_INTE = 20 SMRY_ITER = 2000 SAVE_WEIGHTS_INTE = 4000

SUMMARY_PATH = ROOT_PATH + '/output/summary' TEST_SAVE_PATH = ROOT_PATH + '/tools/test_result'

if NET_NAME.startswith("resnet"): weights_name = NET_NAME elif NET_NAME.startswith("MobilenetV2"): weights_name = "mobilenet/mobilenet_v2_1.0_224" else: raise Exception('net name must in [resnet_v1_101, resnet_v1_50, MobilenetV2]')

PRETRAINED_CKPT = ROOT_PATH + '/data/pretrained_weights/' + weights_name + '.ckpt' TRAINED_CKPT = os.path.join(ROOT_PATH, 'output/trained_weights')

EVALUATE_DIR = ROOT_PATH + '/output/evaluate_result_pickle/'/home/xianyun/cbw/RetinaNet_Tensorflow_Rotation-master/tools/test_dota

EVALUATE_DIR = ROOT_PATH + '/tools/test_dota/'

[yangxue0827]

那可能安装的是cpu版本的tensorflow，应该安装tensorflow-gpu。你check一下gpu有没有被调用以及tensorflow的版本。

[bonnie-cbw]

好的！我检查一下！非常感谢！！！

[bonnie-cbw]

确实是因为我安装的cpu版本的tf！这个问题已经解决了。非常感谢您的解答！ 我还有一个问题想请教一下 我在训练过程中，经常会报以下的错误：

2021-06-28 15:16:43.519534: W tensorflow/core/framework/op_kernel.cc:1389] Unknown: IndexError: index -9223372036853286991 is out of bounds for axis 0 with size 1681218 Traceback (most recent call last):

File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 207, in call ret = func(*args)

File "../libs/detection_oprations/anchor_target_layer_without_boxweight.py", line 38, in anchor_target_layer max_overlaps = overlaps[np.arange(overlaps.shape[0]), argmax_overlaps_inds]

IndexError: index -9223372036853286991 is out of bounds for axis 0 with size 1681218

2021-06-28 15:16:43.684349: W tensorflow/core/kernels/queue_base.cc:277] _0_get_batch/input_producer: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684390: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684484: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684499: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684528: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684546: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684563: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684580: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684598: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684607: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684612: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684632: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684638: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684655: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684664: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684671: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed 2021-06-28 15:16:43.684688: W tensorflow/core/kernels/queue_base.cc:277] _1_get_batch/batch/padding_fifo_queue: Skipping cancelled enqueue attempt with queue not closed Traceback (most recent call last): File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.UnknownError: IndexError: index -9223372036853286991 is out of bounds for axis 0 with size 1681218 Traceback (most recent call last):

File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 207, in call ret = func(*args)

File "../libs/detection_oprations/anchor_target_layer_without_boxweight.py", line 38, in anchor_target_layer max_overlaps = overlaps[np.arange(overlaps.shape[0]), argmax_overlaps_inds]

IndexError: index -9223372036853286991 is out of bounds for axis 0 with size 1681218

 [[{{node tower_1/build_loss/PyFunc}}]]
 [[{{node tower_0/add_3}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "multi_gpu_train.py", line 354, in train() File "multi_gpu_train.py", line 317, in train sess.run([train_op, global_step, total_loss_dict]) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.UnknownError: IndexError: index -9223372036853286991 is out of bounds for axis 0 with size 1681218 Traceback (most recent call last):

File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 207, in call ret = func(*args)

File "../libs/detection_oprations/anchor_target_layer_without_boxweight.py", line 38, in anchor_target_layer max_overlaps = overlaps[np.arange(overlaps.shape[0]), argmax_overlaps_inds]

IndexError: index -9223372036853286991 is out of bounds for axis 0 with size 1681218

 [[node tower_1/build_loss/PyFunc (defined at ../libs/networks/build_whole_network.py:233) ]]
 [[node tower_0/add_3 (defined at multi_gpu_train.py:232) ]]

Caused by op 'tower_1/build_loss/PyFunc', defined at: File "multi_gpu_train.py", line 354, in train() File "multi_gpu_train.py", line 206, in train gpu_id=i) File "../libs/networks/build_whole_network.py", line 233, in build_whole_detection_network tf.float32]) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func return func(*args, *kwargs) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 468, in py_func func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 282, in _internal_py_func input=inp, token=token, Tout=Tout, name=name) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/ops/gen_script_ops.py", line 151, in py_func "PyFunc", input=input, token=token, Tout=Tout, name=name) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(args, **kwargs) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

UnknownError (see above for traceback): IndexError: index -9223372036853286991 is out of bounds for axis 0 with size 1681218 Traceback (most recent call last):

File "/home/xianyun/anaconda3/envs/rretinanet/lib/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 207, in call ret = func(*args)

File "../libs/detection_oprations/anchor_target_layer_without_boxweight.py", line 38, in anchor_target_layer max_overlaps = overlaps[np.arange(overlaps.shape[0]), argmax_overlaps_inds]

IndexError: index -9223372036853286991 is out of bounds for axis 0 with size 1681218

 [[node tower_1/build_loss/PyFunc (defined at ../libs/networks/build_whole_network.py:233) ]]
 [[node tower_0/add_3 (defined at multi_gpu_train.py:232) ]]

想问问您见过这个错误吗？您知道有什么解决的办法吗？ 再次感谢！！！