Closed ooofest closed 1 year ago
I closed various apps and services, such that 51GB of RAM was available, which enabled loading the blip2-opt-6.7b-coco model without any errors.
Summary: there are no free lunches here, you need enough RAM to load any desired models.
Hello, I decided to try the
blip2-opt-6.7b-coco
interrogator and it downloaded a large dataset
when it then tried to run on my directory of files, it apparently ran out of memory:
File "C:\Users\user\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\linear.py", line 96, in init self.weight = Parameter(torch.empty((out_features, in_features), **factory_kwargs)) RuntimeError: [enforce fail at ..\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 268435456 bytes.
I have 64 GB of system RAM and over 30GB free, so this made me wonder if my GPU was being used. To that point, I have two RTX-3090 cards (24GB VRAM each), but that obviously wouldn't work for a larger memory need without some sort of active memory manager.
Is there some sort of additional headroom needed for the memory allocation?
Thanks.