Closed qbwdp closed 6 years ago
If I train mscnn_8s_768_trainval model in second stage, it will crash as following : I0323 16:15:44.452236 18769 sgd_solver.cpp:106] Iteration 81875, lr = 0.0005 I0323 16:17:07.957013 18769 proposal_target_layer.cpp:181] sampled rois: 51, random rois: 77 F0323 16:17:07.964038 18782 math_functions.cu:79] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered Check failure stack trace: F0323 16:17:07.964064 18769 math_functions.cu:79] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered Check failure stack trace: @ 0x7fef502c2daa (unknown) @ 0x7fef502c2daa (unknown) @ 0x7fef502c2ce4 (unknown) @ 0x7fef502c2ce4 (unknown) @ 0x7fef502c26e6 (unknown) @ 0x7fef502c26e6 (unknown) @ 0x7fef502c5687 (unknown) @ 0x7fef502c5687 (unknown) @ 0x7fef50acf078 caffe::caffe_gpu_memcpy() @ 0x7fef50acf078 caffe::caffe_gpu_memcpy() @ 0x7fef5091ce1e caffe::SyncedMemory::to_gpu() @ 0x7fef5091c870 caffe::SyncedMemory::cpu_data() @ 0x7fef5091c2c9 caffe::SyncedMemory::gpu_data() @ 0x7fef5091db12 caffe::Blob<>::cpu_data() @ 0x7fef5091dda2 caffe::Blob<>::gpu_data()
I think ROIPooling has illegal memory access, how to solve this problem?
If I train mscnn_8s_768_trainval model in second stage, it will crash as following : I0323 16:15:44.452236 18769 sgd_solver.cpp:106] Iteration 81875, lr = 0.0005 I0323 16:17:07.957013 18769 proposal_target_layer.cpp:181] sampled rois: 51, random rois: 77 F0323 16:17:07.964038 18782 math_functions.cu:79] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered Check failure stack trace: F0323 16:17:07.964064 18769 math_functions.cu:79] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered Check failure stack trace: @ 0x7fef502c2daa (unknown) @ 0x7fef502c2daa (unknown) @ 0x7fef502c2ce4 (unknown) @ 0x7fef502c2ce4 (unknown) @ 0x7fef502c26e6 (unknown) @ 0x7fef502c26e6 (unknown) @ 0x7fef502c5687 (unknown) @ 0x7fef502c5687 (unknown) @ 0x7fef50acf078 caffe::caffe_gpu_memcpy() @ 0x7fef50acf078 caffe::caffe_gpu_memcpy() @ 0x7fef5091ce1e caffe::SyncedMemory::to_gpu() @ 0x7fef5091c870 caffe::SyncedMemory::cpu_data() @ 0x7fef5091c2c9 caffe::SyncedMemory::gpu_data() @ 0x7fef5091db12 caffe::Blob<>::cpu_data() @ 0x7fef5091dda2 caffe::Blob<>::gpu_data()
I think ROIPooling has illegal memory access, how to solve this problem?