Neither CUDA_VISIBLE_DEVICES nor --cuda-device seems to be respected

[csimokat]

Trying to run:

CUDA_VISIBLE_DEVICES=1 python main.py

and

export CUDA_VISIBLE_DEVICES=1
python main.py

Gets ignored entirely, but checking with: python main.py --help

I see the flag --cuda-devices yet this also does not seem to behave as expected:

$ python main.py --cuda-device 1
**Set cuda device to: 1**
Total VRAM 11009 MB, total RAM 128484 MB
xformers version: 0.0.23.post1
Set vram state to: NORMAL_VRAM
**Device: cuda:0** NVIDIA GeForce RTX 2080 Ti : cudaMallocAsync
VAE dtype: torch.float32
Using xformers cross attention
Starting server

To see the GUI go to: http://127.0.0.1:8188/

So even though I set it to use device 1 it is actually using device 0. This leads to OOM errors since device 0 is busy as illustrated by nvidia-smi and is why I am trying to get this to run on device 1.

$ nvidia-smiWed Dec 27 19:22:28 2023       +-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0     | |-------------------------------+----------------------+----------------------+ | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC | | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. | |                               |                      |               MIG M. | |===============================+======================+======================| |   0  NVIDIA GeForce ...  Off  | 00000000:17:00.0 Off |                  N/A | | 41%   50C    P2    68W / 260W |   1928MiB / 11264MiB |      8%      Default | |                               |                      |                  N/A | +-------------------------------+----------------------+----------------------+ |   1  NVIDIA GeForce ...  Off  | 00000000:65:00.0 Off |                  N/A | | 41%   46C    P8    27W / 260W |     18MiB / 11264MiB |      0%      Default | |                               |                      |                  N/A | +-------------------------------+----------------------+----------------------+                                                                                 +-----------------------------------------------------------------------------+ | Processes:                                                                  | |  GPU   GI   CI        PID   Type   Process name                  GPU Memory | |        ID   ID                                                   Usage      | |=============================================================================| |    0   N/A  N/A      1594      G   /usr/lib/xorg/Xorg                  4MiB | |    0   N/A  N/A   1256382      C   python                           1920MiB | |    1   N/A  N/A      1594      G   /usr/lib/xorg/Xorg                  9MiB | |    1   N/A  N/A      1691      G   /usr/bin/gnome-shell                4MiB | +-----------------------------------------------------------------------------+

$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy

I setup the environment in Anaconda using:

conda create -n sdcomfy python=3.10
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
conda install -c xformers xformers
conda install -c conda-forge transformers
conda install jupyter scikit-learn

I hit the error when trying to execute the SVD example. The SD image prompting works despite the constrained memory on device 0.

[Luxcium]

oh I do have the same problem I am unsure if I am the only one. Someone suggested to hardcore the value in the args... but even without that it does show the selection ok but then the one used is not the same:

Set cuda device to: 1 Total VRAM 12187 MB, total RAM 128490 MB xformers version: 0.0.22 Set vram state to: NORMAL_VRAM Device: cuda:0 NVIDIA TITAN Xp COLLECTORS EDITION : cudaMallocAsync