jacobnrohan
commented
1 month ago I'm getting a similar warning, but after some investigating I can confirm it is running on GPU, despite the error. It may be that this error message is unrelated to the lack of GPU utilization. @alonadi - which version of CUDA do you have? did you install PyTorch and Torchvision accordingly? Here are wheels for pytorch and torchvision for various version of python/CUDA.
Here's what I got:
$CONDA/envs/apeStyleGAN/lib/python3.11/site-packages/torch/autograd/graph.py:768: UserWarning: Attempting to run cuBLAS, but there was no current CUDA context! Attempting to set the primary context... (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:135.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward passThe default network is training at 1.7 second per iteration. nvidia-smi shows ~4GB memory utilization:
| 0 N/A N/A 426695 C ...$CONDA/envs/$MYENV/bin/python 4014MiB |Conda environment.
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
aim 3.23.0 pypi_0 pypi
aim-ui 3.23.0 pypi_0 pypi
aimrecords 0.0.7 pypi_0 pypi
aimrocks 0.5.2 pypi_0 pypi
aiofiles 24.1.0 pypi_0 pypi
alembic 1.13.2 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
anyio 4.4.0 pypi_0 pypi
base58 2.0.1 pypi_0 pypi
boto3 1.34.146 pypi_0 pypi
botocore 1.34.146 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6
ca-certificates 2024.7.2 h06a4308_0
cachetools 5.4.0 pypi_0 pypi
certifi 2024.7.4 pypi_0 pypi
cffi 1.16.0 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
contrastive-learner 0.1.1 pypi_0 pypi
cryptography 43.0.0 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
dnspython 2.6.1 pypi_0 pypi
einops 0.8.0 pypi_0 pypi
email-validator 2.2.0 pypi_0 pypi
fastapi 0.111.1 pypi_0 pypi
fastapi-cli 0.0.4 pypi_0 pypi
filelock 3.15.4 pypi_0 pypi
fire 0.6.0 pypi_0 pypi
fsspec 2024.6.1 pypi_0 pypi
greenlet 3.0.3 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
httpcore 1.0.5 pypi_0 pypi
httptools 0.6.1 pypi_0 pypi
httpx 0.27.0 pypi_0 pypi
idna 3.7 pypi_0 pypi
jinja2 3.1.4 pypi_0 pypi
jmespath 1.0.1 pypi_0 pypi
kornia 0.7.3 pypi_0 pypi
kornia-rs 0.1.5 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.4.4 h6a678d5_1
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h5eee18b_0
mako 1.3.5 pypi_0 pypi
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
ncurses 6.4 h6a678d5_0
networkx 3.3 pypi_0 pypi
numpy 2.0.1 pypi_0 pypi
nvidia-cublas-cu11 11.11.3.6 pypi_0 pypi
nvidia-cuda-cupti-cu11 11.8.87 pypi_0 pypi
nvidia-cuda-nvrtc-cu11 11.8.89 pypi_0 pypi
nvidia-cuda-runtime-cu11 11.8.89 pypi_0 pypi
nvidia-cudnn-cu11 9.1.0.70 pypi_0 pypi
nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi
nvidia-curand-cu11 10.3.0.86 pypi_0 pypi
nvidia-cusolver-cu11 11.4.1.48 pypi_0 pypi
nvidia-cusparse-cu11 11.7.5.86 pypi_0 pypi
nvidia-nccl-cu11 2.20.5 pypi_0 pypi
nvidia-nvtx-cu11 11.8.86 pypi_0 pypi
openssl 1.1.1w h7f8727e_0
packaging 24.1 pypi_0 pypi
pillow 10.4.0 pypi_0 pypi
pip 24.0 py311h06a4308_0
psutil 6.0.0 pypi_0 pypi
py 1.11.0 pypi_0 pypi
pycparser 2.22 pypi_0 pypi
pydantic 2.8.2 pypi_0 pypi
pydantic-core 2.20.1 pypi_0 pypi
pygments 2.18.0 pypi_0 pypi
python 3.11.0 h7a1cb2a_3
python-dateutil 2.9.0.post0 pypi_0 pypi
python-dotenv 1.0.1 pypi_0 pypi
python-multipart 0.0.9 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
readline 8.2 h5eee18b_0
requests 2.32.3 pypi_0 pypi
restrictedpython 7.1 pypi_0 pypi
retry 0.9.2 pypi_0 pypi
rich 13.7.1 pypi_0 pypi
s3transfer 0.10.2 pypi_0 pypi
setuptools 69.5.1 py311h06a4308_0
shellingham 1.5.4 pypi_0 pypi
six 1.16.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
sqlalchemy 2.0.31 pypi_0 pypi
sqlite 3.45.3 h5eee18b_0
starlette 0.37.2 pypi_0 pypi
stylegan2-pytorch 1.8.10 pypi_0 pypi
sympy 1.13.1 pypi_0 pypi
termcolor 2.4.0 pypi_0 pypi
tk 8.6.14 h39e8969_0
torch 2.4.0+cu118 pypi_0 pypi
torchvision 0.19.0+cu118 pypi_0 pypi
tqdm 4.66.4 pypi_0 pypi
triton 3.0.0 pypi_0 pypi
typer 0.12.3 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
tzdata 2024a h04d1e81_0
urllib3 2.2.2 pypi_0 pypi
uvicorn 0.30.3 pypi_0 pypi
uvloop 0.19.0 pypi_0 pypi
vector-quantize-pytorch 0.1.0 pypi_0 pypi
watchfiles 0.22.0 pypi_0 pypi
websockets 12.0 pypi_0 pypi
wheel 0.43.0 py311h06a4308_0
xz 5.4.6 h5eee18b_1
zlib 1.2.13 h5eee18b_1
CUDA details:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
UserWarning: Attempting to run cuBLAS, but there was no current CUDA context! Attempting to set the primary context... (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:135.) return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass.
when I run nvidia-smi, I see that there is not gpu uses.