How to deploy this model with multiple GPUs? - Githubissues

defog-ai / sqlcoder

SoTA LLM for converting natural language questions to SQL queries

Apache License 2.0

3.41k stars 218 forks source link

How to deploy this model with multiple GPUs? #87

Closed LZZJ closed 7 months ago

LZZJ commented 8 months ago

How to deploy this model with multiple GPUs?

There are multiple GPUs on my server. The total size of each GPU is 24GB, but 15GB has already been occupied by other programs.

What modifications need to be made to inference.py? Or which files need to be modified?

github-actions[bot] commented 7 months ago

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] commented 7 months ago

This issue was closed because it has been inactive for 14 days since being marked as stale.