search

snexus / llm-search

Querying local documents, powered by LLM
MIT License
519 stars 60 forks source link

Model params for GPU setup #88

Closed vovsgit closed 4 months ago

vovsgit commented 9 months ago

Hello @nexus! How do I choose the parameters for the models for my GPU setup? I'm manipulate different params, as example n_ctx or n_batch, but is so long, and not effectivity. Maybe do you know formula or method for calculate optimal params for GPU.

I have 2 1080Ti with 11 gb memory, please give me advice ))

snexus commented 9 months ago

Hi,

Sorry, I am not aware of a formula that allows calculating it easily. Would start with monitoring the GPU usage (on Linux, you can use nvidia-smi tool) and optimise the parameters one by one.