Open SaiMadhusudan opened 1 year ago
When I am trying to load the pre trained model I am getting this error
=> loaded checkpoint './3exp/checkpoint.pth.tar' (epoch 100) Load dataset: 2.96 s Compute features 0 / 173 Time: 2.361 (2.361) Cluster the features Faiss assertion 'err == CUBLAS_STATUS_SUCCESS' failed in void faiss::gpu::runMatrixMult(faiss::gpu::Tensor<float, 2, true>&, bool, faiss::gpu::Tensor<T, 2, true>&, bool, faiss::gpu::Tensor<IndexType, 2, true>&, bool, float, float, cublasHandle_t, cudaStream_t) [with AT = float; BT = float; cublasHandle_t = cublasContext; cudaStream_t = CUstream_st] at ./faiss/gpu/utils/MatrixMult-inl.cuh:133; details: cublas failed (13): (512, 256) x (85, 256)' = (512, 85) ./main.sh: line 10: 1379 Aborted (core dumped) python2 main.py ${DIR} --exp ${EXP} --arch ${ARCH} --lr ${LR} --wd ${WD} --k ${K} --sobel --workers ${WORKERS} --epochs 101 --batch 100 --clustering Kmeans --verbose --resume ./3exp/checkpoint.pth.tar Done!
When I am trying to load the pre trained model I am getting this error
=> loaded checkpoint './3exp/checkpoint.pth.tar' (epoch 100) Load dataset: 2.96 s Compute features 0 / 173 Time: 2.361 (2.361) Cluster the features Faiss assertion 'err == CUBLAS_STATUS_SUCCESS' failed in void faiss::gpu::runMatrixMult(faiss::gpu::Tensor<float, 2, true>&, bool, faiss::gpu::Tensor<T, 2, true>&, bool, faiss::gpu::Tensor<IndexType, 2, true>&, bool, float, float, cublasHandle_t, cudaStream_t) [with AT = float; BT = float; cublasHandle_t = cublasContext; cudaStream_t = CUstream_st] at ./faiss/gpu/utils/MatrixMult-inl.cuh:133; details: cublas failed (13): (512, 256) x (85, 256)' = (512, 85) ./main.sh: line 10: 1379 Aborted (core dumped) python2 main.py ${DIR} --exp ${EXP} --arch ${ARCH} --lr ${LR} --wd ${WD} --k ${K} --sobel --workers ${WORKERS} --epochs 101 --batch 100 --clustering Kmeans --verbose --resume ./3exp/checkpoint.pth.tar Done!