KaihuaTang
commented
2 years ago I never encountered this problem before. It’s probably because our workstation has 128G RAM memory. I don’t know how to solve this problem. Sorry about it.
Open Ceazy-Gentleman opened 2 years ago
KaihuaTang
commented
2 years ago I never encountered this problem before. It’s probably because our workstation has 128G RAM memory. I don’t know how to solve this problem. Sorry about it.
MeinhardMark
commented
1 year ago I think I may have the same problem here:
2023-05-22 21:05:02,540 maskrcnn_benchmark INFO: Start evaluation on VG_stanford_filtered_with_attribute_val dataset(5000 images).
0%| | 0/2501 [14:57<?, ?it/s]
Traceback (most recent call last):
File "tools/relation_train_net.py", line 379, in <module>
main()
File "tools/relation_train_net.py", line 372, in main
model = train(cfg, args.local_rank, args.distributed, logger)
File "tools/relation_train_net.py", line 124, in train
run_val(cfg, model, val_data_loaders, distributed, logger)
File "tools/relation_train_net.py", line 259, in run_val
logger=logger,
File "/data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/engine/inference.py", line 110, in inference
predictions = compute_on_dataset(model, data_loader, device, synchronize_gather=cfg.TEST.RELATION.SYNC_GATHER, timer=inference_timer)
File "/data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/engine/inference.py", line 34, in compute_on_dataset
output = model(images.to(device), targets)
File "/data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 50, in forward
proposals, proposal_losses = self.rpn(images, features, targets)
File "/data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/modeling/rpn/rpn.py", line 161, in forward
return self._forward_test(anchors, objectness, rpn_box_regression)
File "/data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/modeling/rpn/rpn.py", line 187, in _forward_test
boxes = self.box_selector_test(anchors, objectness, rpn_box_regression)
File "/data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
result = self.forward(*input, **kwargs)
File "/data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/modeling/rpn/inference.py", line 142, in forward
sampled_boxes.append(self.forward_for_single_feature_map(a, o, b))
File "/data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/modeling/rpn/inference.py", line 122, in forward_for_single_feature_map
score_field="objectness",
File "/data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/structures/boxlist_ops.py", line 28, in boxlist_nms
keep = _box_nms(boxes, score, nms_thresh)
File "/data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/amp/amp.py", line 25, in wrapper
return wrap_fn(orig_fn, inner_cast_fn, handle)(*args, **kwargs)
File "/data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/amp/wrap.py", line 28, in wrapper
return orig_fn(*new_args, **kwargs)
RuntimeError: CUDA error: no kernel image is available for execution on the device (launch_kernel at /tmp/pip-req-build-ufslq_a9/aten/src/ATen/native/cuda/Loops.cuh:103)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x6d (0x7f9265a9b2ad in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: void at::native::gpu_index_kernel<__nv_dl_wrapper_t<__nv_dl_tag<void (*)(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>), &(void at::native::index_kernel_impl<at::native::OpaqueType<8> >(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>)), 1u>> >(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>, __nv_dl_wrapper_t<__nv_dl_tag<void (*)(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>), &(void at::native::index_kernel_impl<at::native::OpaqueType<8> >(at::TensorIterator&, c10::ArrayRef<long>, c10::ArrayRef<long>)), 1u>> const&) + 0x82d (0x7f926b4258ad in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #2: <unknown function> + 0x5720533 (0x7f926b428533 in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #3: <unknown function> + 0x5720948 (0x7f926b428948 in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #4: <unknown function> + 0x13899e2 (0x7f92670919e2 in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #5: at::native::index(at::Tensor const&, c10::ArrayRef<at::Tensor>) + 0x4cf (0x7f926709f1ef in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #6: <unknown function> + 0x19ff5f1 (0x7f92677075f1 in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #7: <unknown function> + 0x19710e7 (0x7f92676790e7 in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #8: <unknown function> + 0x36c7d18 (0x7f92693cfd18 in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #9: <unknown function> + 0x19710e7 (0x7f92676790e7 in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #10: at::Tensor::index(c10::ArrayRef<at::Tensor>) const + 0x1bc (0x7f9288cbf5ac in /data/mmj/anaconda3/envs/sgcu101/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #11: nms_cuda(at::Tensor, float) + 0x824 (0x7f922961e80a in /data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/_C.cpython-37m-x86_64-linux-gnu.so)
frame #12: nms(at::Tensor const&, at::Tensor const&, float) + 0x7ea (0x7f922962978a in /data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/_C.cpython-37m-x86_64-linux-gnu.so)
frame #13: <unknown function> + 0xb57d7 (0x7f92296377d7 in /data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/_C.cpython-37m-x86_64-linux-gnu.so)
frame #14: <unknown function> + 0xb1dd8 (0x7f9229633dd8 in /data/mmj/Scene-Graph-Benchmark.pytorch/maskrcnn_benchmark/_C.cpython-37m-x86_64-linux-gnu.so)
<omitting python frames>
frame #22: python() [0x4ae8df]
frame #26: python() [0x4ae8df]
frame #29: python() [0x4ae8df]
frame #33: python() [0x4c9a80]
frame #39: python() [0x578b67]
frame #41: python() [0x4aea19]
frame #44: python() [0x4ae8df]
frame #48: python() [0x4c9a80]
frame #54: python() [0x578b67]
frame #56: python() [0x4aea19]
frame #60: python() [0x4c9a80]Did you solve yours?
❓ Questions and Help
I encounter another problem that SGDet model of Neural-Motifs (with and without attributes, tried both of them) can not be evaluated. It can be trained ,but failed in Starting validating(Starting evaluation on IMP_sgdet.txt motif_sgdet.txt IMP_sgdet.txt VG_stanford_filtered_with_attribute_val dataset(5000 images)).
there is two problem (1)OSError: [Errno 12] Cannot allocate memory .(2) RuntimeError: DataLoader worker (pid 48413) is killed by signal: Killed.When I searching the problem in google, they said that the proble may be not enough RAM memory(my RAM memory is 32 G, with 2 2080Ti,per gpu with 11G memory) when validation. Do you know how to reduce the memory occupation? I can not find the a command to solve the problem. the attachment is the runing log with motifs and IMP.