search

google-research / scenic

Scenic: A Jax Library for Computer Vision Research and Beyond
Apache License 2.0
3.33k stars 438 forks source link

[OWL-VIT] Out Of Memory error #929

Open yayaja11 opened 1 year ago

yayaja11 commented 1 year ago

Hi

I am using two 2080ti which has 11GB GPU memory

but i think the sample code only use one gpu.

predictions = jitted(

jaxlib.xla_extension.XlaRuntimeError: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 7247809568 bytes. BufferAssignment OOM Debugging. BufferAssignment stats: parameter allocation: 1.81GiB constant allocation: 36.0KiB maybe_live_out allocation: 25.96MiB preallocated temp allocation: 6.75GiB preallocated temp fragmentation: 0B (0.00%) total allocation: 8.58GiB total fragmentation: 15.86MiB (0.18%)

I set my env XLA_PYTHON_CLIENT_MEM_FRACTION=.75 XLA_PYTHON_CLIENT_PREALLOCATE=false

yayaja11 commented 1 year ago

What kind of gpu did you use? And what is the minimum GPU memory required to run it?

toffo10 commented 1 year ago

Hi, did you find a way to fix it? I'm trying to fine tune the b16 model but I get the same error

huangfei00 commented 9 months ago

os.environ["XLA_PYTHON_CLIENT_PREALLOCATE"]="false" os.environ["XLA_PYTHON_CLIENT_MEM_FRACTION"]=".XX" os.environ["XLA_PYTHON_CLIENT_ALLOCATOR"]="platform" try this, it works for me.