No AVX CPU (former mining rig) - workaround (?)

[martindellavecchia]

I think I am dealing with an issue trying to get my LORA trained with my GPUs as my former rig runs a Pentium Gold CPU with no AVX support on it.

Is there any workaround I can use to avoid purchasing a new cpu? I would love to use my rig with a 2080ti and 2 3060s for tranining purposes.

Thanks in advance

[b-fission]

I was about to post an answer on your other thread #2581, but this'll do. I had compiled builds of the tensorflow and bitsandbytes for Windows that can run without AVX.

See here for packages and instructions

[martindellavecchia]

It moved on!, but I faced another issue:

return F.linear(input, self.weight, self.bias)

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 10.00 MiB. GPU 0 has a total capacty of 11.00 GiB of which 0 bytes is free. Of the allocated memory 10.16 GiB is allocated by PyTorch, and 80.42 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF steps: 0%|

weird, as i have a 2080ti, 11GB and a 3060 12GB.

Should I also be updating pytorch?

[b-fission]

I have no experience doing multi-GPU training, so I can only make guesses.

What type of model are you training on? SDXL or SD 1.5? Is your batch size too large? Use gradient checkpointing?

And I think PyTorch 2.1.2 is installed by default on the current version which should be fine.

[martindellavecchia]

it seems to be running now for LORA training, i didn't notice i was on the dreambooth tab when i was failing with above allocation error 👀 . LORA training looks to be working fine, but it just picking up GPU0. I will dig a bit trying to make it work for multi GPU, thanks so much, your tensorflow with no AVX did the trick! ;) 🚀

Thanks so much!