search

PromtEngineer / localGPT-Vision

Chat with your documents using Vision Language Models. This repo implements an End to End RAG pipeline with both local and proprietary VLMs
430 stars 88 forks source link

Not working on 12GB VRAM (RTX 3060) - out of memory on indexing #14

Open luculli opened 1 month ago

luculli commented 1 month ago

Hi, I tried Ubuntu 22.04 with an NVIDIA RTX 3060 (12 GB) and it ran out of memory during the indexing process:

Error indexing files: Error indexing files: CUDA out of memory. Tried to allocate 266.00 MiB. GPU 0 has a total capacity of 11.75 GiB of which 191.56 MiB is free. Including non-PyTorch memory, this process has 10.99 GiB memory in use. Of the allocated memory 10.10 GiB is allocated by PyTorch, and 768.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

image

Do you have any idea about the required minimum VRAM?

ALutz273 commented 2 weeks ago

i have the same problem 

      .;cccccccccccccccccccccc;.         OS: Fedora Linux 40 (Workstation Edition) x86_64 
    .:cccccccccccccccccccccccccc:.       Host: B650M PG Riptide 
  .;ccccccccccccc;.:dddl:.;ccccccc;.     Kernel: 6.11.6-200.fc40.x86_64 
 .:ccccccccccccc;OWMKOOXMWd;ccccccc:.    Uptime: 8 mins 
.:ccccccccccccc;KMMc;cc;xMMc:ccccccc:.   Packages: 2339 (rpm), 20 (flatpak) 
,cccccccccccccc;MMM.;cc;;WW::cccccccc,   Shell: bash 5.2.26 
:cccccccccccccc;MMM.;cccccccccccccccc:   Resolution: 3840x1600 
:ccccccc;oxOOOo;MMM0OOk.;cccccccccccc:   DE: GNOME 46.6 
cccccc:0MMKxdd:;MMMkddc.;cccccccccccc;   WM: Mutter 
ccccc:XM0';cccc;MMM.;cccccccccccccccc'   WM Theme: Adwaita 
ccccc;MMo;ccccc;MMW.;ccccccccccccccc;    Theme: Adwaita [GTK2/3] 
ccccc;0MNc.ccc.xMMd:ccccccccccccccc;     Icons: Adwaita [GTK2/3] 
cccccc;dNMWXXXWM0::cccccccccccccc:,      Terminal: gnome-terminal 
cccccccc;.:odl:.;cccccccccccccc:,.       CPU: AMD Ryzen 9 7950X (32) @ 5.881GHz 
:cccccccccccccccccccccccccccc:'.         GPU: NVIDIA GeForce RTX 4070 Ti 
.:cccccccccccccccccccccc:;,..            GPU: AMD ATI 78:00.0 Raphael 
  '::cccccccccccccc::;,.                 Memory: 5230MiB / 31152MiB