search

nerdyrodent / VQGAN-CLIP

Just playing with getting VQGAN+CLIP running locally, rather than having to use colab.
Other
2.59k stars 428 forks source link

Training with RTX 3090 #118

Closed Amatumm closed 2 years ago

Amatumm commented 2 years ago

Hei all! I am new to this VQGAN and CLIP, so I would like to have some help with it.

I am training this with PC Win10, Anaconda3 and RTX 3090 24Gb (and i9). I have installed all the cudas and gpu drivers. I also train with my other computer PC Win10, Anaconda3 and GTX 1080 8Gb (and i5) (this is only for the reference as seen below).

My no. 1 issue is that when I train with 3090, it uses the GPU's memory VRAM but the load goes to the CPU. With 1080 it uses the GPU VRAM only.

The nvidia-smi finds the GPU and it tells me that it is in a place number 0. In the code -cd number 0 is CPU. I have tried to use numbers 0-3 and 8-9 (according the number of core in my CPU, but they were not recognized. Am I missing something here or am I just being silly not to understand how this works?

I will attach few pictures of this:

Capture4

Capture5

Capture3

Capture6

The other question was that when I tried to train images 900x900 I got the Cuda OutOf Memory -error. I have 24Gb VRAM so that should be ebough, but do I need to close ALL my other programs when doing this? In the error it told me I had oinly 21+Gb of memory in use...

Thank you in advance if someone can hel me with this! And if this question does not belong here, please remove it if needed.

manafoo commented 2 years ago

some gpu load does not display correctly on the task manager , use msi after burner or any othe tools to analyze the workload on your gpu