search

yangxue0827 / R-DFPN_FPN_Tensorflow

R-DFPN: Rotation Dense Feature Pyramid Networks (Tensorflow)
http://www.mdpi.com/2072-4292/10/1/132
120 stars 47 forks source link

CUDA_ERROR_ILLEGAL_ADDRESS error when many objects in one image #5

Open Lizw14 opened 6 years ago

Lizw14 commented 6 years ago

Hi,

I encountered the problem of CUDA_ERROR_ILLEGAL_ADDRESS error during training when the objects are densely located, i.e. when there are many(roughly >100 or 150) ships in one 6001000 image. When I split these dense images into smaller ones(control object number < 50) and pad the splitted images into 6001000, the training goes on well with no error. BTW, the same error happens in all your repositories(R2CNN, RRPN, R-DFPN).

But I think split the images with densely located objects may damage the final performance, so I want to get this problem solved. Could you please tell me which part cause this error? Maybe GPU NMS or something else?

wechatimg7

powermano commented 6 years ago

Thanks for your share. When i control the objects per image, it finally works.

FishBigOcean commented 5 years ago

Hi,

I encountered the problem of CUDA_ERROR_ILLEGAL_ADDRESS error during training when the objects are densely located, i.e. when there are many(roughly >100 or 150) ships in one 600_1000 image. When I split these dense images into smaller ones(control object number < 50) and pad the splitted images into 600_1000, the training goes on well with no error. BTW, the same error happens in all your repositories(R2CNN, RRPN, R-DFPN).

But I think split the images with densely located objects may damage the final performance, so I want to get this problem solved. Could you please tell me which part cause this error? Maybe GPU NMS or something else?

wechatimg7

Thanks for sharing！ Did you finally find a solution other than cropping the picture? look forward to your reply

zsh1234567 commented 4 years ago

May I ask what dataset set do you use?