search

wafflecomposite / langchain-ask-pdf-local

An AI-app that allows you to upload a PDF and ask questions about it. It uses StableVicuna 13B and runs locally.
87 stars 8 forks source link

I have gpu and I expect to run model faster, but your code is only for cpu? how to change it? #4

Open alexhmyang opened 1 year ago

alexhmyang commented 1 year ago

as for the quesiton: Requirements GPU is not used and is not required.

I have gpu and I expect to run model faster, but your code is only for cpu? how to change it?

wafflecomposite commented 1 year ago

Until I make some updates, check out this fork https://github.com/sebaxzero/LangChain_PDFChat_Oobabooga

I haven't tried it myself, but looks like it should be capable to utilize GPU.

LebToki commented 1 year ago

You could try this approach! Haven't tested it yet though

GPU Based Approach

llm = LlamaCpp(
    model_path="./models/stable-vicuna-13B.ggmlv3.q8_0.bin",
    stop=["### Human:"],
    callback_manager=callback_manager,
    verbose=True,
    n_ctx=2048,
    n_batch=512,
    device="cuda"  # Specify the GPU device (e.g., "cuda:0")
)