search

gpgpu-sim / gpgpu-sim_distribution

GPGPU-Sim provides a detailed simulation model of contemporary NVIDIA GPUs running CUDA and/or OpenCL workloads. It includes support for features such as TensorCores and CUDA Dynamic Parallelism as well as a performance visualization tool, AerialVisoin, and an integrated energy model, GPUWattch.
Other
1.1k stars 505 forks source link

RuntimeError: cublas runtime error : library not initialized at #177

Open HamHyungkyu opened 4 years ago

HamHyungkyu commented 4 years ago

Hello, I'm trying to run a simple pytorch code on pytorch-gpgpu-sim, but I met this problem..

x = torch.randn(10,10, device="cuda:0")
y = torch.randn(10,10, device="cuda:0")
z = torch.matmul(x, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: cublas runtime error : library not initialized at  /home/user/pytorch-gpgpu-sim/aten/src/THC/THCGeneral.cpp:333

Also I met this kind of problem in different code. RuntimeError: CuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Does anyone know why these problems occur?

My env is centos 7 python 3.6.9 gcc 5.4.0 CUDA 8.0 CUDNN 7.1 PYTORCH_BIN = /home/user/pytorch-gpgpu-sim/torch/lib/libcaffe2_gpu.so TITANV config

cng123 commented 4 years ago

Hi,

Could you try running the above code on hardware with a fully updated pytorch (reinstall from pip)? It may be possible that one of the solutions in https://github.com/torch/cutorch/issues/677 could fix the issue.

If the above does not fix the problem, could you attach the full output log of the script? Also, if possible, could you post your environmental variables, as well as the output log from building pytorch?

Wen-Tian-Pineapple commented 11 months ago

Hello, I'm trying to run a simple pytorch code on pytorch-gpgpu-sim, but I met this problem..

x = torch.randn(10,10, device="cuda:0")
y = torch.randn(10,10, device="cuda:0")
z = torch.matmul(x, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: cublas runtime error : library not initialized at  /home/user/pytorch-gpgpu-sim/aten/src/THC/THCGeneral.cpp:333

Also I met this kind of problem in different code. RuntimeError: CuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Does anyone know why these problems occur?

My env is centos 7 python 3.6.9 gcc 5.4.0 CUDA 8.0 CUDNN 7.1 PYTORCH_BIN = /home/user/pytorch-gpgpu-sim/torch/lib/libcaffe2_gpu.so TITANV config

@HamHyungkyu Hello, Did you find a way to bypass this issue? I had the same library not initialized problem and couldn't fix it, I tried link static library but I got some other bugs. Thanks!