Open dancingpipi opened 4 years ago
Hi z13974509906,
Thanks for reaching out! I would like to try and reproduce this issue.
Do you mind sharing the model that you were attempting to convert?
Best, John
Hi z13974509906,
Thanks for reaching out! I would like to try and reproduce this issue.
Do you mind sharing the model that you were attempting to convert?
Best, John
I have follow pytorch yolov3 . I use torch2trt convert the backbone. After convert, pytorch became abnormal as I mention: b[a] will cause an indexerror, so I have to change it to b[a.tolist()].
If I don't convert the backbone by torch2trt, b[a] work properly
Hi z13974509906,
Thanks for reaching out! I would like to try and reproduce this issue.
Do you mind sharing the model that you were attempting to convert?
Best, John
you can also test torchvison.models.resnet.resnet50()
, it can reproduce this issue
any update?
IndexError: too many indices for tensor of dimension 2 Hello i also found the same problem, may i have any solution
IndexError: too many indices for tensor of dimension 2 Hello i also found the same problem, may i have any solution
Uss tensor.tolist() may solve it. But it won't work everywhere
Hello where is it placed?for example x=randn(1,3,244,244).cuda().tolist(), Can you tell me exactly where to change? thanks
Hello where is it placed?for example x=randn(1,3,244,244).cuda().tolist(), Can you tell me exactly where to change? thanks
no, I found it only impact index. for example : b[a] should change to b[a.tolist()]
Hello, do you have an email? I have some questions for you. Thank you
I have the same problem, add tolist() can solve it, But I don't know why?
the same problem, add tolist() can solve it, But I don't kno
Can you tell me where to add tolist (), I didn't solve the problem,thanks!
the same problem, add tolist() can solve it, But I don't kno
Can you tell me where to add tolist (), I didn't solve the problem,thanks!
add tolist() in your index tensor array
the same problem, add tolist() can solve it, But I don't kno
Can you tell me where to add tolist (), I didn't solve the problem,thanks!
I may be powerless. The only way to solve the problem is debugging step by step.
I recommend pdb or code
I think this is the same as https://github.com/NVIDIA-AI-IOT/torch2trt/issues/270
I have the same problem. I debug the code on Nvidia jetson AGX, after debugging, I think torch2trt will really impact the original pytorch. Can anyone solve it? Or tell me the reason for the problem. Thanks!
@dancingpipi @PressEtoRace Did you succeed in optimizing inference time of YOLOv3? if yes, Do you detect all objects well with a good accuracy? because the model I am working on can detect only one object!
@dancingpipi @PressEtoRace Did you succeed in optimizing inference time of YOLOv3? if yes, Do you detect all objects well with a good accuracy? because the model I am working on can detect only one object! I've had problems like yours. I think this is also caused by the influence of torch2trt on pytorch. My solution is to modify the NMS code in yolov3. You can print the size of the data after each operation in NMS, and you should find the problem.
same problem
I have also found out when running mask rcnn with mmdetection, it would throw cuda error at torch.tensor.any()
RuntimeError: CUDA error: an illegal memory access was encountered
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: an illegal memory access was encountered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f24d0f949ab in /home/acer/nfs-share/pytorch/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xc10 (0x7f24d11d7280 in /home/acer/nfs-share/pytorch/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f24d0f7ce0d in /home/acer/nfs-share/pytorch/torch/lib/libc10.so)
frame #3: <unknown function> + 0x559d82 (0x7f24ed5eed82 in /home/acer/nfs-share/pytorch/torch/lib/libtorch_python.so)
<omitting python frames>
frame #19: __libc_start_main + 0xe7 (0x7f2538809b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #20: _start + 0x2a (0x56250f80577a in /home/acer/.pyenv/versions/pytorch_build/bin/python)
but after commenting out the converter of getitem (https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/torch2trt/converters/getitem.py#L32), it would work fine.
I think overriding torch.getitem is dangerous, since other torch api would use that. why not convert torch.narrow
(https://pytorch.org/docs/stable/tensors.html)
I have also found out when running mask rcnn with mmdetection, it would throw cuda error at torch.tensor.any()
RuntimeError: CUDA error: an illegal memory access was encountered terminate called after throwing an instance of 'c10::Error' what(): CUDA error: an illegal memory access was encountered Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f24d0f949ab in /home/acer/nfs-share/pytorch/torch/lib/libc10.so) frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xc10 (0x7f24d11d7280 in /home/acer/nfs-share/pytorch/torch/lib/libc10_cuda.so) frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f24d0f7ce0d in /home/acer/nfs-share/pytorch/torch/lib/libc10.so) frame #3: <unknown function> + 0x559d82 (0x7f24ed5eed82 in /home/acer/nfs-share/pytorch/torch/lib/libtorch_python.so) <omitting python frames> frame #19: __libc_start_main + 0xe7 (0x7f2538809b97 in /lib/x86_64-linux-gnu/libc.so.6) frame #20: _start + 0x2a (0x56250f80577a in /home/acer/.pyenv/versions/pytorch_build/bin/python)
but after commenting out the converter of getitem (https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/torch2trt/converters/getitem.py#L32), it would work fine.
I think overriding torch.getitem is dangerous, since other torch api would use that. why not convert
torch.narrow
(https://pytorch.org/docs/stable/tensors.html)
would you please show how to solve this problem more clearly? thanks!
Tried this example with PR #691 and there is no error.
in pytorch, I can do the below code right:
but if I use torch2trt to convert model, then execute the code. error whill occur:
IndexError: too many indices for tensor of dimension 4
Will any one reproduce this error.
Wait for your help~