search

zhoushengisnoob / DeepClustering

Methods and Implements of Deep Clustering
2.78k stars 414 forks source link

VADE Model : CUDA error: CUBLAS_STATUS_EXECUTION_FAILED #41

Closed Amalsalem closed 9 months ago

Amalsalem commented 12 months ago

I have an error while runing my model m it is related to CUDA

File /data/software/miniconda3/lib/python3.9/site-packages/torch/nn/modules/linear.py:114, in Linear.forward(self, input) 113 def forward(self, input: Tensor) -> Tensor: --> 114 return F.linear(input, self.weight, self.bias)

RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasLtMatmul( ltHandle, computeDesc.descriptor(), &alpha_val, mat1_ptr, Adesc.descriptor(), mat2_ptr, Bdesc.descriptor(), &beta_val, result_ptr, Cdesc.descriptor(), result_ptr, Cdesc.descriptor(), &heuristicResult.algo, workspace.data_ptr(), workspaceSize, at::cuda::getCurrentCUDAStream())

0%| | 0/10 [00:00<?, ?it/s]/data/software/miniconda3/lib/python3.9/site-packages/torch/optim/lr_scheduler.py:138: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate warnings.warn("Detected call of lr_scheduler.step() before optimizer.step(). "

zhoushengisnoob commented 9 months ago

Please see my answer in #43 issue:)