search

chengdazhi / Deformable-Convolution-V2-PyTorch

Deformable ConvNets V2 (DCNv2) in PyTorch
MIT License
1.44k stars 229 forks source link

error in deformable_col2im: invalid argument #13

Closed jwwangchn closed 5 years ago

jwwangchn commented 5 years ago

I met an error: error in deformable_col2im: invalid argument when I run my program, and the program can continue running. But I didn't meet this error when I use the other dataset whose image size is (342, 342).

gasvn commented 5 years ago

I met a similar error. When the input image is larger than 1024x1024 (1, 512, 128, 128 for that layer), the same invalid argument error occurs. I think the reason for this error is that the pre-allocated memory space is not enough. But I not familiar with the Cuda program in pytorch, so I can't find a solution yet.

happycoding1996 commented 5 years ago

Same problem. Pytorch 0.4.0, python 3.5, cuda 8.0.

chengdazhi commented 5 years ago

Hi, could you provide a short code snippet to reproduce the bug? Thanks a lot!

chengdazhi commented 5 years ago

FYI, my environment is PyTorch 0.4.1, python 3.6/3.7, cuda 9.0.

shaoqb commented 5 years ago

Similar error，how to solve it? Pytorch 0.4.1, python 3.7, cuda 9.0

chengdazhi commented 5 years ago

Possible cause: channel dimension too large.

You can try to change the src/deform_conv_cuda_kernel.cu at line No.68 to

const int CUDA_NUM_THREADS = 1024; const int kMaxGridNum = 65535; inline int GET_BLOCKS(const int N) {   return std::min(kMaxGridNum, (N + CUDA_NUM_THREADS - 1) / CUDA_NUM_THREADS); }

and see if this can solve the problem.  

jwwangchn commented 5 years ago

@chengdazhi It solved my problem. Thanks.

chengdazhi commented 5 years ago

@jwwangchn Thanks for your feedback. I have updated the .cu file in the commit 50a282 .

abhijay9 commented 4 years ago

Hi,

I have set up the deform conv on a machine with a single 1060 and it works but when I tried using the same on a machine with (multi-gpu) GTX TITAN X it gives me an error:

error in modulated_deformable_im2col_cuda: invalid device function

This is already present in the file deform_conv_cuda_kernel.cu https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/issues/13#issuecomment-472084619

What do you think might be the problem?

MurrayC7 commented 4 years ago

Hi,

I have set up the deform conv on a machine with a single 1060 and it works but when I tried using the same on a machine with (multi-gpu) GTX TITAN X it gives me an error:

error in modulated_deformable_im2col_cuda: invalid device function

This is already present in the file deform_conv_cuda_kernel.cu #13 (comment)

What do you think might be the problem?

@chengdazhi @abhijay-g Did you figure it out? I also met this issue.