search

defog-ai / sqlcoder

SoTA LLM for converting natural language questions to SQL queries
Apache License 2.0
3.44k stars 219 forks source link

torch.cuda.OutOfMemoryError: CUDA out of memory. #44

Closed 8188 closed 8 months ago

8188 commented 1 year ago

I use 7b-model with 3090(24G), When loading, it cost 15G Memory, It's OK. When referencing, indeed it is a nice model, but it eat too much Memory as metadata.sql increasing a bit. Only a 4k metadata.sql probably causes CUDA out of memory. Any good idea to solve it?

github-actions[bot] commented 10 months ago

Stale issue message

deqncho2 commented 10 months ago

I'm interested in the same question. How do we feed in large schemas (I'm talking > 10k) without running out of memory on inference?

github-actions[bot] commented 9 months ago

This issue is stale because it has been open for 30 days with no activity.

fire717 commented 9 months ago

Same question here.

dqatsh commented 9 months ago

try

load_in_4bit=True & num_beams=3

it works for me with 4090(24G)

github-actions[bot] commented 8 months ago

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] commented 8 months ago

This issue was closed because it has been inactive for 14 days since being marked as stale.