Model params for GPU setup

snexus / llm-search

Querying local documents, powered by LLM

MIT License

421 stars 51 forks source link

Model params for GPU setup #88

Closed vovsgit closed 2 days ago

vovsgit commented 5 months ago

Hello @nexus! How do I choose the parameters for the models for my GPU setup? I'm manipulate different params, as example n_ctx or n_batch, but is so long, and not effectivity. Maybe do you know formula or method for calculate optimal params for GPU.

I have 2 1080Ti with 11 gb memory, please give me advice ))

snexus commented 4 months ago

Hi,

Sorry, I am not aware of a formula that allows calculating it easily. Would start with monitoring the GPU usage (on Linux, you can use nvidia-smi tool) and optimise the parameters one by one.