wenxi2
commented
9 hours ago sys.platform linux
Python 3.8.17 (default, Jul 5 2023, 21:04:15) [GCC 11.2.0]
numpy 1.24.3
detectron2 0.6 @/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/detectron2
Compiler GCC 7.3
CUDA compiler CUDA 11.1
detectron2 arch flags 3.7, 5.0, 5.2, 6.0, 6.1, 7.0, 7.5, 8.0, 8.6
DETECTRON2_ENV_MODULE
PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.1
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
- CuDNN 8.0.5
- Magma 2.5.2
[WARNING]your gpu arch (8, 6) isn't compiled in prebuilt, may cause invalid device function. available: {(6, 1), (3, 7), (7, 0), (5, 0), (6, 0), (7, 5), (5, 2)} [Exception|indice_conv|subm]feat=torch.Size([100432, 256]),w=torch.Size([3, 3, 256, 256]),pair=torch.Size([2, 9, 100432]),pairnum=tensor([95659, 98114, 95665, 97724, 0, 0, 0, 0, 0], dtype=torch.int32),act=100432,algo=ConvAlgo.Native SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue. x <spconv.pytorch.core.SparseConvTensor object at 0x7f730398bb20> [WARNING]your gpu arch (8, 6) isn't compiled in prebuilt, may cause invalid device function. available: {(6, 1), (3, 7), (7, 0), (5, 0), (6, 0), (7, 5), (5, 2)} [Exception|indice_conv|subm]feat=torch.Size([109840, 256]),w=torch.Size([3, 3, 256, 256]),pair=torch.Size([2, 9, 109840]),pairnum=tensor([105367, 107530, 105370, 107432, 0, 0, 0, 0, 0], dtype=torch.int32),act=109840,algo=ConvAlgo.Native SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue. Traceback (most recent call last): File "infer_visdrone.py", line 16, in
launch(
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/detectron2/engine/launch.py", line 67, in launch
mp.spawn(
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
while not context.join():
File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 1 terminated with the following error: Traceback (most recent call last): File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, args) File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/detectron2/engine/launch.py", line 126, in _distributed_worker main_func(args) File "/home/abab/QueryDet-PyTorch-main/train_tools/visdrone_infer.py", line 170, in start_train res = Trainer.test(cfg, model) File "/home/abab/QueryDet-PyTorch-main/train_tools/visdrone_infer.py", line 77, in test result = inference_on_dataset(model, data_loader, evaluator) File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/detectron2/evaluation/evaluator.py", line 158, in inference_on_dataset outputs = model(inputs) File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, kwargs) File "/home/abab/QueryDet-PyTorch-main/models/querydet/detector.py", line 172, in forward return self.test(batched_inputs) File "/home/abab/QueryDet-PyTorch-main/models/querydet/detector.py", line 289, in test results, total_time = self.test_forward(images) # normal test File "/home/abab/QueryDet-PyTorch-main/models/querydet/detector.py", line 357, in test_forward det_cls_query, det_bbox_query, query_anchors = self.qInfer.run_qinfer(params, CAS_feature, features_value, File "/home/abab/QueryDet-PyTorch-main/models/querydet/qinfer.py", line 226, in run_qinfer cls_result_i = self._run_spconvs(x, self.cls_spconv).view(-1, self.anchor_num self.num_classes)[inds] File "/home/abab/QueryDet-PyTorch-main/models/querydet/qinfer.py", line 201, in _run_spconvs y = filters(x) File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, kwargs) File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/pytorch/modules.py", line 137, in forward input = module(input) File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, *kwargs) File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/pytorch/conv.py", line 330, in forward out_features = Fsp.indice_subm_conv( File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 216, in decorate_fwd return fwd(args, **kwargs) File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/pytorch/functional.py", line 294, in forward raise e File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/pytorch/functional.py", line 279, in forward return ops.indice_conv(features, File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/pytorch/ops.py", line 695, in indice_conv tuned_res, min_time = GEMM.tune_and_cache( File "/usr/local/anaconda3/envs/querydet/lib/python3.8/site-packages/spconv/algo.py", line 358, in tune_and_cache GemmMainUnitTest.matmul2(params) RuntimeError: /io/build/temp.linux-x86_64-cpython-38/spconv/build/src/cumm/gemm/main/GemmMainUnitTest/GemmMainUnitTest_matmul2.cu 4489 cuda execution failed with error 209 no kernel image is available for execution on the device Simt_f32f32f32f32f32tnt_m64n256k8m32n64k8_2_SAC00 error with params [100432, 256] [256, 256] [100432, 256]
Could you please help me with this problem? Thank you.