search

illuin-tech / colpali

The code used to train and run inference with the ColPali architecture.
https://huggingface.co/vidore
MIT License
850 stars 75 forks source link

High memory usage #87

Closed iamkhalidbashir closed 1 week ago

iamkhalidbashir commented 1 week ago

Is this normal for 1 pdf file with 15 pages for vidore/colqwen2-v0.1 model

CUDA out of memory. Tried to allocate 59.39 GiB. GPU 0 has a total capacity of 23.55 GiB of which 11.38 GiB is free. Process 1712302 has 12.16 GiB memory in use. Of the allocated memory 11.25 GiB is allocated by PyTorch, and 473.74 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
ManuelFay commented 1 week ago

No it's not normal

iamkhalidbashir commented 1 week ago

Does cuda version has to do something with it?

ManuelFay commented 1 week ago

Are you batching your documents little by little ? Check out the example inference code with dataloaders to batch docs 4 by 4 and then move to CPU - I guess that may be the mistake

iamkhalidbashir commented 1 week ago

Yes by batching useing DataLoader it worked

May I know how to convert the embeddings back to Image or base64